Stochastic gradient estimate variance in contrastive divergence and persistent contrastive divergence

Mathias Berglund

Stochastic gradient estimate variance in contrastive divergence and persistent contrastive divergence

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

1 Citation (Scopus)

Abstract

Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and variances for stochastic gradient estimates of individual data points. It is well known that CD yields a biased gradient estimate. In this paper we however show empirically that CD has a lower stochastic gradient estimate variance than unbiased sampling, while the mean of subsequent PCD estimates has a higher variance than independent sampling. The results give one explanation to the finding that CD can be used with smaller minibatches or higher learning rates than PCD.

Original language	English
Title of host publication	ESANN 2016 - 24th European Symposium on Artificial Neural Networks
Pages	521-526
Number of pages	6
ISBN (Electronic)	978-2-87578-027-8
Publication status	Published - 1 Jan 2016
MoE publication type	A4 Article in a conference publication
Event	European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning - Bruges, Belgium Duration: 27 Apr 2016 → 29 Apr 2016 Conference number: 24

Conference

Conference	European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning
Abbreviated title	ESANN
Country	Belgium
City	Bruges
Period	27/04/2016 → 29/04/2016

Access to Document

Link to publication in Scopus

Fingerprint Dive into the research topics of 'Stochastic gradient estimate variance in contrastive divergence and persistent contrastive divergence'. Together they form a unique fingerprint.

Cite this

@inproceedings{2570ea5eb8584e5580abb74c354faaae,

title = "Stochastic gradient estimate variance in contrastive divergence and persistent contrastive divergence",

abstract = "Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and variances for stochastic gradient estimates of individual data points. It is well known that CD yields a biased gradient estimate. In this paper we however show empirically that CD has a lower stochastic gradient estimate variance than unbiased sampling, while the mean of subsequent PCD estimates has a higher variance than independent sampling. The results give one explanation to the finding that CD can be used with smaller minibatches or higher learning rates than PCD.",

author = "Mathias Berglund",

year = "2016",

month = jan,

day = "1",

language = "English",

isbn = "978-2-87587-026-1",

pages = "521--526",

booktitle = "ESANN 2016 - 24th European Symposium on Artificial Neural Networks",

note = " European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN ; Conference date: 27-04-2016 Through 29-04-2016",

}

TY - GEN

T1 - Stochastic gradient estimate variance in contrastive divergence and persistent contrastive divergence

AU - Berglund, Mathias

N1 - Conference code: 24

PY - 2016/1/1

Y1 - 2016/1/1

N2 - Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and variances for stochastic gradient estimates of individual data points. It is well known that CD yields a biased gradient estimate. In this paper we however show empirically that CD has a lower stochastic gradient estimate variance than unbiased sampling, while the mean of subsequent PCD estimates has a higher variance than independent sampling. The results give one explanation to the finding that CD can be used with smaller minibatches or higher learning rates than PCD.

AB - Contrastive Divergence (CD) and Persistent Contrastive Divergence (PCD) are popular methods for training Restricted Boltzmann Machines. However, both methods use an approximate method for sampling from the model distribution. As a side effect, these approximations yield significantly different biases and variances for stochastic gradient estimates of individual data points. It is well known that CD yields a biased gradient estimate. In this paper we however show empirically that CD has a lower stochastic gradient estimate variance than unbiased sampling, while the mean of subsequent PCD estimates has a higher variance than independent sampling. The results give one explanation to the finding that CD can be used with smaller minibatches or higher learning rates than PCD.

UR - http://www.scopus.com/inward/record.url?scp=84994086889&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84994086889

SN - 978-2-87587-026-1

SP - 521

EP - 526

BT - ESANN 2016 - 24th European Symposium on Artificial Neural Networks

T2 - European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning

Y2 - 27 April 2016 through 29 April 2016

ER -