Both the construction and use of buildings cause significant environmental pressures. The greenhouse gas (GHG) emissions imposed by buildings have been studied rather extensively, but less is known about other impacts. Still, climate change is only one harmful impact driven by buildings. Furthermore, no studies exist about how the other impacts are correlated with GHG emissions in the building context, and thus to what extent GHGs could be utilized as a more general environmental performance indicator. This paper fills these gaps by presenting a life cycle assessment of the pre-use phase of a modern concrete-element residential building with a very comprehensive life cycle inventory (LCI). The focus of the study is on the comparison of the accumulation of different environmental impacts relative to GHGs. The accumulation is analyzed from two perspectives common to building LCAs: building systems and different construction materials. The ReCiPe midpoint assessment method is utilized to reach wide impact category coverage. The study shows how GHGs act as a relatively good indicator for eight impact categories, but not for the others. The study also depicts that a very high coverage in the LCI must be reached to capture the majority of the different impacts. Many materials and building systems are considered non-relevant and are often excluded from building LCAs, which are in fact of great importance in many impact categories.