Sequential Inference for Latent Temporal Gaussian Process Models

During the recent decades Gaussian processes (GPs) have become increasingly popular tools for non-parametric Bayesian data analysis in a wide range of applications, including non-linear regression, classification, spatial statistics, dynamic system modeling and non-linear dimension reduction. The flexibility of Gaussian processes lies in their covariance functions, which encode the prior beliefs about the latent function to be modeled in different applications. While GPs are useful models with successful real-world applications, they face several practical problems in modeling and inference. Firstly, the posterior inference on the latent variables is computationally intensive as it scales cubically in the number of data points and is analytically intractable in all but the Gaussian measurement noise case. Secondly, construction of new models where Gaussian processes are used as latent components often requires analytic work (such as the derivation of the covariance function), which can be hard or even impossible in practice. In this work, we apply Gaussian processes to model stochastic dynamic systems. The main aim of this thesis is to rekindle an old idea of converting a GP prior with a given covariance function into an equivalent state-space model commonly used for modeling dynamic systems. The state-space form of GP priors has several advantages over the traditional covariance function view. Firstly, one can perform posterior inference with sequential algorithms which usually scale linearly in the number of data points. Secondly, construction of new latent Gaussian process is easy as one does not need to derive any covariance functions. We show how the conversion can be done for several important covariance functions, and present practical algorithms that are suited for the resulting state-space models. In addition to one-dimensional Gaussian processes, we shall consider the conversion of more general spatio-temporal Gaussian processes as well as latent force models, where Gaussian processes are used to model unknown forces acting on mechanistic dynamic systems. The second aim of this thesis is to develop new inference methods for general state-space models. In particular, we develop new methods for filtering and smoothing various non-linear state-space models as well as for estimating parameters of non-linear stochastic differential equations.