Rethinking Inference in Gaussian Processes: A Dual Parameterization Approach

Uncertainty quantification is a vital aspect of machine learning, especially when accurate estimates of uncertainty are crucial for making informed decisions. Gaussian Processes (GPs), known for their versatility as function space priors, find wide-ranging applications in diverse fields, including climate modelling and epidemiology. GPs are particularly useful due to their non-linearity, allowing them to adapt to various data patterns and their ability to integrate domain-specific knowledge. As probabilistic models, they offer predictions and quantify the uncertainty within these predictions, an essential feature in scenarios demanding reliable forecasts.  This thesis focuses on applying GPs to large-scale, non-Gaussian sequential data. Due to their non-parametric nature, GPs face increasing computational demands as data size grows. The requirement for a conjugate Gaussian likelihood for computational tractability presents further challenges. Therefore, it is common to use approximate inference for applying GPs to non-Gaussian likelihoods alongside scalable model formulations to handle complex data distributions in real- world applications.  The theme connecting the papers is an innovative approach to parameterizing the optimization problem in approximate inference, centring on a forgotten parameterization termed the dual parameters. This fresh perspective offers methods to enhance the efficiency of GPs in applying them to large and complex datasets, particularly in the context of sequential data. This approach addresses the pivotal challenges of tractability and scalability inherent in GPs in the sequential setting. The concept of dual parameters serves as a unifying framework, linking all approximate inference techniques through their various likelihood approximations.  In addition, the thesis shows the application of dual parameterization methods in a range of GP model formulations and problem settings. It introduces a new algorithm for inference and learning in non-Gaussian time series data and the sparse GP framework. The applications discussed extend to areas such as Bayesian optimization and continual learning, highlighting the adaptability and potential of GPs in contemporary machine learning.