Bayesian Hierarchical Stacking: Some Models Are (Somewhere) Useful

Yuling Yao; Gregor Pirš; Aki Vehtari; Andrew Gelman

doi:10.1214/21-BA1287

Bayesian Hierarchical Stacking: Some Models Are (Somewhere) Useful

Yuling Yao^*
, Gregor Pirš
, Aki Vehtari
, Andrew Gelman

^*Corresponding author for this work

Research output: Contribution to journal › Article › Scientific › peer-review

29 Citations (Scopus)

293 Downloads (Pure)

Abstract

Stacking is a widely used model averaging technique that asymptotically yields optimal predictions among linear averages. We show that stacking is most effective when model predictive performance is heterogeneous in inputs, and we can further improve the stacked mixture with a hierarchical model. We generalize stacking to Bayesian hierarchical stacking. The model weights are varying as a function of data, partially-pooled, and inferred using Bayesian inference. We further incorporate discrete and continuous inputs, other structured priors, and time series and longitudinal data. To verify the performance gain of the proposed method, we derive theory bounds, and demonstrate on several applied problems.

Original language	English
Pages (from-to)	1043-1071
Number of pages	29
Journal	Bayesian Analysis
Volume	17
Issue number	4
DOIs	https://doi.org/10.1214/21-BA1287
Publication status	Published - Dec 2022
MoE publication type	A1 Journal article-refereed

Funding

∗The authors thank the National Science Foundation, Institute of Education Sciences, Office of Naval Research, National Institutes of Health, Sloan Foundation, Schmidt Futures, and the Academy of Finland Flagship programme: Finnish Center for Artificial Intelligence (FCAI) for partial financial support. Gregor Pirˇs is supported by the Slovenian Research Agency young researcher grant. †Flatiron Institute, New York, USA, [email protected] ‡Faculty of Computer and Information Science, University of Ljubljana, Ljubljana, Slovenia §Department of Computer Science, Aalto University, Espoo, Finland ¶Department of Statistics and Political Science, Columbia University, New York, USA

Keywords

Bayesian hierarchical modeling
Conditional prediction
Covariate shift
Model averaging
Prior construction
Stacking

Access to Document

10.1214/21-BA1287Licence: CC BY

Bayesian Hierarchical StackingFinal published version, 1.08 MBLicence: CC BY

Cite this

@article{26431c04d548402e98c8c8ea3c265a96,

title = "Bayesian Hierarchical Stacking: Some Models Are (Somewhere) Useful",

abstract = "Stacking is a widely used model averaging technique that asymptotically yields optimal predictions among linear averages. We show that stacking is most effective when model predictive performance is heterogeneous in inputs, and we can further improve the stacked mixture with a hierarchical model. We generalize stacking to Bayesian hierarchical stacking. The model weights are varying as a function of data, partially-pooled, and inferred using Bayesian inference. We further incorporate discrete and continuous inputs, other structured priors, and time series and longitudinal data. To verify the performance gain of the proposed method, we derive theory bounds, and demonstrate on several applied problems.",

keywords = "Bayesian hierarchical modeling, Conditional prediction, Covariate shift, Model averaging, Prior construction, Stacking",

author = "Yuling Yao and Gregor Pir{\v s} and Aki Vehtari and Andrew Gelman",

note = "Publisher Copyright: {\textcopyright} 2022 International Society for Bayesian Analysis",

year = "2022",

month = dec,

doi = "10.1214/21-BA1287",

language = "English",

volume = "17",

pages = "1043--1071",

journal = "Bayesian Analysis",

issn = "1936-0975",

publisher = "International Society for Bayesian Analysis",

number = "4",

}

TY - JOUR

T1 - Bayesian Hierarchical Stacking: Some Models Are (Somewhere) Useful

AU - Yao, Yuling

AU - Pirš, Gregor

AU - Vehtari, Aki

AU - Gelman, Andrew

PY - 2022/12

Y1 - 2022/12

N2 - Stacking is a widely used model averaging technique that asymptotically yields optimal predictions among linear averages. We show that stacking is most effective when model predictive performance is heterogeneous in inputs, and we can further improve the stacked mixture with a hierarchical model. We generalize stacking to Bayesian hierarchical stacking. The model weights are varying as a function of data, partially-pooled, and inferred using Bayesian inference. We further incorporate discrete and continuous inputs, other structured priors, and time series and longitudinal data. To verify the performance gain of the proposed method, we derive theory bounds, and demonstrate on several applied problems.

AB - Stacking is a widely used model averaging technique that asymptotically yields optimal predictions among linear averages. We show that stacking is most effective when model predictive performance is heterogeneous in inputs, and we can further improve the stacked mixture with a hierarchical model. We generalize stacking to Bayesian hierarchical stacking. The model weights are varying as a function of data, partially-pooled, and inferred using Bayesian inference. We further incorporate discrete and continuous inputs, other structured priors, and time series and longitudinal data. To verify the performance gain of the proposed method, we derive theory bounds, and demonstrate on several applied problems.

KW - Bayesian hierarchical modeling

KW - Conditional prediction

KW - Covariate shift

KW - Model averaging

KW - Prior construction

KW - Stacking

UR - https://www.scopus.com/pages/publications/85139205468

U2 - 10.1214/21-BA1287

DO - 10.1214/21-BA1287

M3 - Article

AN - SCOPUS:85139205468

SN - 1936-0975

VL - 17

SP - 1043

EP - 1071

JO - Bayesian Analysis

JF - Bayesian Analysis

IS - 4

ER -

Bayesian Hierarchical Stacking: Some Models Are (Somewhere) Useful

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this