Expectation Propagation as a Way of Life: A Framework for Bayesian Inference on Partitioned Data

Aki Vehtari*, Andrew Gelman, Tuomas Sivula, Pasi Jylanki, Dustin Tran, Swupnil Sahai, Paul Blomstedt, John P. Cunningham, David Schiminovich, Christian P. Robert

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

55 Downloads (Pure)

Abstract

A common divide-and-conquer approach for Bayesian computation with big data is to partition the data, perform local inference for each piece separately, and combine the results to obtain a global posterior approximation. While being conceptually and computationally appealing, this method involves the problematic need to also split the prior for the local inferences; these weakened priors may not provide enough regularization for each separate computation, thus eliminating one of the key advantages of Bayesian methods. To resolve this dilemma while still retaining the generalizability of the underlying local inference method, we apply the idea of expectation propagation (EP) as a framework for distributed Bayesian inference. The central idea is to iteratively update approximations to the local likelihoods given the state of the other approximations and the prior.

The present paper has two roles: we review the steps that are needed to keep EP algorithms numerically stable, and we suggest a general approach, inspired by EP, for approaching data partitioning problems in a way that achieves the computational benefits of parallelism while allowing each local update to make use of relevant information from the other sites. In addition, we demonstrate how the method can be applied in a hierarchical context to make use of partitioning of both data and parameters. The paper describes a general algorithmic framework, rather than a specific algorithm, and presents an example implementation for it.

Original languageEnglish
Pages (from-to)1-53
Number of pages53
JournalJournal of Machine Learning Research
Volume21
Publication statusPublished - 2020
MoE publication typeA1 Journal article-refereed

Keywords

  • Bayesian computation
  • data partitioning
  • expectation propagation
  • hierarchical models
  • statistical computing
  • PRECISION MATRIX
  • CLASSIFICATION
  • LIKELIHOOD
  • REGRESSION
  • MODELS

Equipment

Science-IT

Mikko Hakala (Manager)

School of Science

Facility/equipment: Facility

  • Cite this