Manifold Mixup: Better Representations by Interpolating Hidden States

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Researchers

  • Vikas Verma

  • Alex Lamb
  • Christopher Beckham
  • Amir Najafi
  • Ioannis Mitliagkas
  • David Lopez-Paz
  • Yoshua Bengio

Research units

  • Montreal Institute for Learning Algorithms
  • Sharif University of Technology
  • Facebook Artificial Intelligence Research

Abstract

Deep neural networks excel at learning the training data, but often provide incorrect and confident predictions when evaluated on slightly different test examples. This includes distribution shifts, outliers, and adversarial examples. To address these issues, we propose Manifold Mixup, a simple regularizer that encourages neural networks to predict less confidently on interpolations of hidden representations. Manifold Mixup leverages semantic interpolations as additional training signal, obtaining neural networks with smoother decision boundaries at multiple levels of representation. As a result, neural networks trained with Manifold Mixup learn class-representations with fewer directions of variance. We prove theory on why this flattening happens under ideal conditions, validate it on practical situations, and connect it to previous works on information theory and generalization. In spite of incurring no significant computation and being implemented in a few lines of code, Manifold Mixup improves strong baselines in supervised learning, robustness to single-step adversarial attacks, and test log-likelihood.

Details

Original languageEnglish
Title of host publicationProceedings of the 36th International Conference on Machine Learning
Publication statusPublished - 2019
MoE publication typeA4 Article in a conference publication
EventInternational Conference on Machine Learning - Long Beach, United States
Duration: 9 Jun 201915 Jun 2019
Conference number: 36

Publication series

NameProceedings of Machine Learning Research
PublisherPMLR
Volume97
ISSN (Electronic)6438-6447

Conference

ConferenceInternational Conference on Machine Learning
Abbreviated titleICML
CountryUnited States
CityLong Beach
Period09/06/201915/06/2019

    Research areas

  • Deep Learning

ID: 38544178