Ladder Variational Autoencoders

Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, Ole Winther

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review


Variational autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models. We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models. We provide a detailed analysis of the learned hierarchical latent representation and show that our new inference model is qualitatively different and utilizes a deeper more distributed hierarchy of latent variables. Finally, we observe that batch-normalization and deterministic warm-up (gradually turning on the KL-term) are crucial for training variational models with many stochastic layers.
Original languageEnglish
Title of host publicationAdvances in Neural Information Processing Systems
PublisherNeural Information Processing Systems Foundation
Number of pages9
Publication statusPublished - 2016
MoE publication typeA4 Article in a conference publication
EventIEEE Conference on Neural Information Processing Systems - Barcelona, Spain
Duration: 5 Dec 201610 Dec 2016
Conference number: 30

Publication series

NameAdvances in neural information processing systems
PublisherNeural Information Processing Systems Foundation
ISSN (Print)1049-5258


ConferenceIEEE Conference on Neural Information Processing Systems
Abbreviated titleNIPS

Fingerprint Dive into the research topics of 'Ladder Variational Autoencoders'. Together they form a unique fingerprint.

  • Cite this

    Kaae Sønderby, C., Raiko, T., Maaløe, L., Kaae Sønderby, S., & Winther, O. (2016). Ladder Variational Autoencoders. In Advances in Neural Information Processing Systems (pp. 3745-3753). (Advances in neural information processing systems; Vol. 29). Neural Information Processing Systems Foundation.