Memory-Based Dual Gaussian Processes for Sequential Learning

Paul E. Chang; Prakhar Verma; S.T. John; Arno Solin; Mohammad Emtiyaz Khan

Memory-Based Dual Gaussian Processes for Sequential Learning

Paul E. Chang, Prakhar Verma, S.T. John, Arno Solin, Mohammad Emtiyaz Khan

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

49 Downloads (Pure)

Abstract

Sequential learning with Gaussian processes (GPs) is challenging when access to past data is limited, for example, in continual and active learning. In such cases, errors can accumulate over time due to inaccuracies in the posterior, hyperparameters, and inducing points, making accurate learning challenging. Here, we present a method to keep all such errors in check using the recently proposed dual sparse variational GP. Our method enables accurate inference for generic likelihoods and improves learning by actively building and updating a memory of past data. We demonstrate its effectiveness in several applications involving Bayesian optimization, active learning, and continual learning.

Original language	English
Title of host publication	Proceedings of the 40th International Conference on Machine Learning
Editors	Andread Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, Jonathan Scarlett
Publisher	JMLR
Pages	4035-4054
Number of pages	20
Publication status	Published - Jul 2023
MoE publication type	A4 Conference publication
Event	International Conference on Machine Learning - Honolulu, United States Duration: 23 Jul 2023 → 29 Jul 2023 Conference number: 40

Publication series

Name	Proceedings of Machine Learning Research
Publisher	PMLR
Volume	202
ISSN (Electronic)	2640-3498

Conference

Conference	International Conference on Machine Learning
Abbreviated title	ICML
Country/Territory	United States
City	Honolulu
Period	23/07/2023 → 29/07/2023

Access to Document

SCI_Chang_etal_ICML_2023Final published version, 5.5 MBLicence: CC BY

https://proceedings.mlr.press/v202/chang23a.htmlLicence: CC BY

1 Active
2 Finished

Solin Arno /AoF Fellow Salary: Probabilistic principles for latent space exploration in deep learning
Solin, A. (Principal investigator) & Mereu, R. (Project Member)
01/09/2021 → 31/08/2026
Project: Academy of Finland: Other research funding
-: Shallow models meet deep vision
Solin, A. (Principal investigator), Wang, H. (Project Member), Tamir, E. (Project Member), Trapp, M. (Project Member), Verma, P. (Project Member), Chang, P. (Project Member), Li, R. (Project Member) & Mereu, R. (Project Member)
01/09/2019 → 31/08/2023
Project: Academy of Finland: Other research funding
-: Finnish Center for Artificial Intelligence
Kaski, S. (Principal investigator)
01/01/2019 → 31/12/2022
Project: Academy of Finland: Other research funding

Science-IT
Hakala, M. (Manager)
School of Science
Facility/equipment: Facility

Cite this

Chang, P. E., Verma, P., John, S. T., Solin, A., & Emtiyaz Khan, M. (2023). Memory-Based Dual Gaussian Processes for Sequential Learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the 40th International Conference on Machine Learning (pp. 4035-4054). (Proceedings of Machine Learning Research; Vol. 202). JMLR. https://proceedings.mlr.press/v202/chang23a.html

Chang, Paul E. ; Verma, Prakhar ; John, S.T. et al. / Memory-Based Dual Gaussian Processes for Sequential Learning. Proceedings of the 40th International Conference on Machine Learning. editor / Andread Krause ; Emma Brunskill ; Kyunghyun Cho ; Barbara Engelhardt ; Sivan Sabato ; Jonathan Scarlett. JMLR, 2023. pp. 4035-4054 (Proceedings of Machine Learning Research).

@inproceedings{c1741c38c5974f978341cd81f6017ab5,

title = "Memory-Based Dual Gaussian Processes for Sequential Learning",

abstract = "Sequential learning with Gaussian processes (GPs) is challenging when access to past data is limited, for example, in continual and active learning. In such cases, errors can accumulate over time due to inaccuracies in the posterior, hyperparameters, and inducing points, making accurate learning challenging. Here, we present a method to keep all such errors in check using the recently proposed dual sparse variational GP. Our method enables accurate inference for generic likelihoods and improves learning by actively building and updating a memory of past data. We demonstrate its effectiveness in several applications involving Bayesian optimization, active learning, and continual learning. ",

author = "Chang, {Paul E.} and Prakhar Verma and S.T. John and Arno Solin and {Emtiyaz Khan}, Mohammad",

year = "2023",

month = jul,

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "JMLR",

pages = "4035--4054",

editor = "Andread Krause and Emma Brunskill and Kyunghyun Cho and Barbara Engelhardt and Sivan Sabato and Jonathan Scarlett",

booktitle = "Proceedings of the 40th International Conference on Machine Learning",

address = "United States",

note = "International Conference on Machine Learning, ICML ; Conference date: 23-07-2023 Through 29-07-2023",

}

Chang, PE , Verma, P , John, ST , Solin, A & Emtiyaz Khan, M 2023, Memory-Based Dual Gaussian Processes for Sequential Learning. in A Krause, E Brunskill, K Cho, B Engelhardt, S Sabato & J Scarlett (eds), Proceedings of the 40th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 202, JMLR, pp. 4035-4054, International Conference on Machine Learning, Honolulu, Hawaii, United States, 23/07/2023. <https://proceedings.mlr.press/v202/chang23a.html>

Memory-Based Dual Gaussian Processes for Sequential Learning. / Chang, Paul E.; Verma, Prakhar ; John, S.T. et al.
Proceedings of the 40th International Conference on Machine Learning. ed. / Andread Krause; Emma Brunskill; Kyunghyun Cho; Barbara Engelhardt; Sivan Sabato; Jonathan Scarlett. JMLR, 2023. p. 4035-4054 (Proceedings of Machine Learning Research; Vol. 202).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - Memory-Based Dual Gaussian Processes for Sequential Learning

AU - Chang, Paul E.

AU - Verma, Prakhar

AU - John, S.T.

AU - Solin, Arno

AU - Emtiyaz Khan, Mohammad

N1 - Conference code: 40

PY - 2023/7

Y1 - 2023/7

N2 - Sequential learning with Gaussian processes (GPs) is challenging when access to past data is limited, for example, in continual and active learning. In such cases, errors can accumulate over time due to inaccuracies in the posterior, hyperparameters, and inducing points, making accurate learning challenging. Here, we present a method to keep all such errors in check using the recently proposed dual sparse variational GP. Our method enables accurate inference for generic likelihoods and improves learning by actively building and updating a memory of past data. We demonstrate its effectiveness in several applications involving Bayesian optimization, active learning, and continual learning.

AB - Sequential learning with Gaussian processes (GPs) is challenging when access to past data is limited, for example, in continual and active learning. In such cases, errors can accumulate over time due to inaccuracies in the posterior, hyperparameters, and inducing points, making accurate learning challenging. Here, we present a method to keep all such errors in check using the recently proposed dual sparse variational GP. Our method enables accurate inference for generic likelihoods and improves learning by actively building and updating a memory of past data. We demonstrate its effectiveness in several applications involving Bayesian optimization, active learning, and continual learning.

M3 - Conference article in proceedings

T3 - Proceedings of Machine Learning Research

SP - 4035

EP - 4054

BT - Proceedings of the 40th International Conference on Machine Learning

A2 - Krause, Andread

A2 - Brunskill, Emma

A2 - Cho, Kyunghyun

A2 - Engelhardt, Barbara

A2 - Sabato, Sivan

A2 - Scarlett, Jonathan

PB - JMLR

T2 - International Conference on Machine Learning

Y2 - 23 July 2023 through 29 July 2023

ER -

Memory-Based Dual Gaussian Processes for Sequential Learning

Abstract

Publication series

Conference

Access to Document

Fingerprint

Projects

Solin Arno /AoF Fellow Salary: Probabilistic principles for latent space exploration in deep learning

-: Shallow models meet deep vision

-: Finnish Center for Artificial Intelligence

Equipment

Science-IT

Cite this