Foundations and Advances in Deep Learning

Kyunghyun Cho

    Research output: ThesisDoctoral ThesisCollection of Articles


    Deep neural networks have become increasingly popular under the name of deep learning recently due to their success in challenging machine learning tasks. Although the popularity is mainly due to recent successes, the history of neural networks goes as far back as 1958 when Rosenblatt presented a perceptron learning algorithm. Since then, various kinds of artificial neural networks have been proposed. They include Hopfield networks, self-organizing maps, neural principal component analysis, Boltzmann machines, multi-layer perceptrons, radial-basis function networks, autoencoders, sigmoid belief networks, support vector machines and deep belief networks. The first part of this thesis investigates shallow and deep neural networks in search of principles that explain why deep neural networks work so well across a range of applications. The thesis starts from some of the earlier ideas and models in the field of artificial neural networks and arrive at autoencoders and Boltzmann machines which are two most widely studied neural networks these days. The author thoroughly discusses how those various neural networks are related to each other and how the principles behind those networks form a foundation for autoencoders and Boltzmann machines. The second part is the collection of the ten recent publications by the author. These publications mainly focus on learning and inference algorithms of Boltzmann machines and autoencoders. Especially, Boltzmann machines, which are known to be difficult to train, have been in the main focus. Throughout several publications the author and the co-authors have devised and proposed a new set of learning algorithms which includes the enhanced gradient, adaptive learning rate and parallel tempering. These algorithms are further applied to a restricted Boltzmann machine with Gaussian visible units. In addition to these algorithms for restricted Boltzmann machines the author proposed a two-stage pretraining algorithm that initializes the parameters of a deep Boltzmann machine to match the variational posterior distribution of a similarly structured deep autoencoder. Finally, deep neural networks are applied to image denoising and speech recognition.
    Translated title of the contributionFoundations and Advances in Deep Learning
    Original languageEnglish
    QualificationDoctor's degree
    Awarding Institution
    • Aalto University
    • Karhunen, Juha, Supervising Professor
    • Raiko, Tapani, Thesis Advisor
    • Ilin, Alexander, Thesis Advisor
    Print ISBNs978-952-60-5574-9
    Electronic ISBNs978-952-60-5575-6
    Publication statusPublished - 2014
    MoE publication typeG5 Doctoral dissertation (article)


    • deep learning
    • neural networks
    • multilayer perceptron
    • probabilistic model
    • restricted Boltzmann machine
    • deep Boltzmann machine
    • denoising autoencoder


    Dive into the research topics of 'Foundations and Advances in Deep Learning'. Together they form a unique fingerprint.

    Cite this