Information bottlenecks and dimensionality reduction in deep learning: Notes on speech processing

Press/Media: Social media activity

Description

Autoencoders and other deep neural networks with information bottlenecks have become fashionable. The heuristic idea is that the dimensionality of the hidden layers is reduced such that the network is forced to focus on the important part of the data. Experiments have also demonstrated that autoencoders are efficient in this sense. I have however been left wondering whether the amount of information can be characterized in exact terms. How much information flows through the bottleneck? How would we even measure that?

Period3 Sept 2020

Media contributions

1

Media contributions

  • TitleInformation bottlenecks and dimensionality reduction in deep learning: Notes on speech processing
    Media name/outletMedium
    Media typeWeb
    Date03/09/2020
    DescriptionAutoencoders and other deep neural networks with information bottlenecks have become fashionable. The heuristic idea is that the dimensionality of the hidden layers is reduced such that the network is forced to focus on the important part of the data. Experiments have also demonstrated that autoencoders are efficient in this sense. I have however been left wondering whether the amount of information can be characterized in exact terms. How much information flows through the bottleneck? How would we even measure that?
    URLhttps://towardsdatascience.com/information-bottlenecks-c2ee67015065
    PersonsTom Bäckström