Energy-based Latent Aligner for Incremental Learning

K. J. Joseph*, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, Vineeth N. Balasubramanian

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

15 Citations (Scopus)


Deep learning models tend to forget their earlier knowledge while incrementally learning new tasks. This behavior emerges because the parameter updates optimized for the new tasks may not align well with the updates suitable for older tasks. The resulting latent representation mismatch causes forgetting. In this work, we propose ELI: Energy-based Latent Aligner for Incremental Learning, which first learns an energy manifold for the latent representations such that previous task latents will have low energy and the current task latents have high energy values. This learned manifold is used to counter the representational shift that happens during incremental learning. The implicit regularization that is offered by our proposed methodology can be used as a plug-and-play module in existing incremental learning methodologies. We validate this through extensive evaluation on CIFAR-100, ImageNet subset, ImageNet 1k and Pascal VOC datasets. We observe consistent improvement when ELI is added to three prominent methodologies in class-incremental learning, across multiple incremental settings. Further, when added to the state-of-the-art incremental object detector, ELI provides over 5% improvement in detection accuracy, corroborating its effectiveness and complementary advantage to the existing art. Code is available at:

Original languageEnglish
Title of host publication2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Number of pages10
ISBN (Electronic)978-1-6654-6946-3
Publication statusPublished - 2022
MoE publication typeA4 Conference publication
EventIEEE Conference on Computer Vision and Pattern Recognition - New Orleans, United States
Duration: 18 Jun 202224 Jun 2022

Publication series

NameIEEE Computer Society Conference on Computer Vision and Pattern Recognition
ISSN (Print)1063-6919


ConferenceIEEE Conference on Computer Vision and Pattern Recognition
Abbreviated titleCVPR
Country/TerritoryUnited States
CityNew Orleans


Dive into the research topics of 'Energy-based Latent Aligner for Incremental Learning'. Together they form a unique fingerprint.

Cite this