A new method is proposed for estimation of lower bounds for the magnitudes of as-built deviations against the as-designed Building Information Model (BIM) given a single spherical panoramic image of the scene. The method is based on deforming the as-designed BIM so that its projection onto the image fits optimally with edge features extracted from the image. The lower bounds derived are equal to the magnitudes of approximately minimal movements of BIM vertices that explain the observed image. A novel approach based on region growing along edge pixels is proposed for extraction of arcs on the image sphere corresponding to BIM lines. A method based on line to line correspondences is extended to arc to line correspondences to solve the exterior orientation of the spherical panorama with a novel approach for translation estimation and an adaptive weighting in rotation estimation. Testing with synthetic data and real imagery captured at three different sites shows that major deviations can be detected and lower bounds for the magnitudes of the deviations estimated from a single image only. The method is able to correctly handle adjacent or occluded correspondences located close to each other. The method is shown to be more than nine times faster than a comparison point cloud-based method.