Vibration-based structural health monitoring is based on detecting changes in the dynamic characteristics of the structure. It is well known that environmental or operational variations can also have an influence on the vibration properties. If these effects are not taken into account, they can result in false indications of damage. If the environmental or operational variations cause nonlinear effects, they can be compensated using a Gaussian mixture model (GMM) without the measurement of the underlying variables. The number of Gaussian components can also be estimated. For the local linear components, minimum mean square error (MMSE) estimation is applied to eliminate the environmental or operational influences. Damage is detected from the residuals after applying principal component analysis (PCA). Control charts are used for novelty detection. The proposed approach is validated using simulated data and the identified lowest natural frequencies of the Z24 Bridge under temperature variation. Nonlinear models are most effective if the data dimensionality is low. On the other hand, linear models often outperform nonlinear models for high-dimensional data.