Two-layer contractive encodings with shortcuts for semi-supervised learning

Hannes Schulz, Kyunghyun Cho, Tapani Raiko, Sven Behnke

    Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

    2 Citations (Scopus)

    Abstract

    Supervised training of multi-layer perceptrons (MLP) with only few labeled examples is prone to overfitting. Pretraining an MLP with unlabeled samples of the input distribution may achieve better generalization. Usually, pretraining is done in a layer-wise, greedy fashion which limits the complexity of the learnable features. To overcome this limitation, two-layer contractive encodings have been proposed recently - which pose a more difficult optimization problem, however. On the other hand, linear transformations of perceptrons have been proposed to make optimization of deep networks easier. In this paper, we propose to combine these two approaches. Experiments on handwritten digit recognition show the benefits of our combined approach to semi-supervised learning.

    Original languageEnglish
    Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Pages450-457
    Number of pages8
    Volume8226 LNCS
    EditionPART 1
    DOIs
    Publication statusPublished - 2013
    MoE publication typeA4 Article in a conference publication
    EventInternational Conference on Neural Information Processing - Daegu, Korea, Republic of
    Duration: 3 Nov 20137 Nov 2013
    Conference number: 20

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    NumberPART 1
    Volume8226 LNCS
    ISSN (Print)03029743
    ISSN (Electronic)16113349

    Conference

    ConferenceInternational Conference on Neural Information Processing
    Abbreviated titleICONIP
    CountryKorea, Republic of
    CityDaegu
    Period03/11/201307/11/2013

    Keywords

    • Linear transformation
    • Multi-layer perceptron
    • Semi-supervised learning
    • Two-layer contractive encoding

    Fingerprint

    Dive into the research topics of 'Two-layer contractive encodings with shortcuts for semi-supervised learning'. Together they form a unique fingerprint.

    Cite this