Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU

Erik Schultheis; Rohit Babbar

doi:10.1007/978-3-031-43418-1_41

Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU

^*Corresponding author for this work

University of Bath

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

2 Citations (Scopus)

13 Downloads (Pure)

Abstract

In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied naïvely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases
Subtitle of host publication	Research Track - European Conference, ECML PKDD 2023, Proceedings
Editors	Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, Francesco Bonchi
Publisher	Springer
Pages	689-704
Number of pages	16
ISBN (Print)	978-3-031-43417-4
DOIs	https://doi.org/10.1007/978-3-031-43418-1_41
Publication status	Published - 2023
MoE publication type	A4 Conference publication
Event	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - Turin, Italy Duration: 18 Sept 2023 → 22 Sept 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Publisher	Springer
Volume	14171 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases
Abbreviated title	ECML PKDD
Country/Territory	Italy
City	Turin
Period	18/09/2023 → 22/09/2023

Access to Document

10.1007/978-3-031-43418-1_41Licence: CC BY

SCI_Schultheis_etal_ECML_PKDD_2023Final published version, 307 KBLicence: CC BY

Cite this

Schultheis, E., & Babbar, R. (2023). Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. In D. Koutra, C. Plant, M. Gomez Rodriguez, E. Baralis, & F. Bonchi (Eds.), Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings (pp. 689-704). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14171 LNAI). Springer. https://doi.org/10.1007/978-3-031-43418-1_41

Schultheis, Erik ; Babbar, Rohit. / Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. editor / Danai Koutra ; Claudia Plant ; Manuel Gomez Rodriguez ; Elena Baralis ; Francesco Bonchi. Springer, 2023. pp. 689-704 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{543dff7049444cddbe8a6df3365b6557,

title = "Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU",

abstract = "In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied na{\"i}vely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse.",

author = "Erik Schultheis and Rohit Babbar",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD ; Conference date: 18-09-2023 Through 22-09-2023",

year = "2023",

doi = "10.1007/978-3-031-43418-1_41",

language = "English",

isbn = "978-3-031-43417-4",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "689--704",

editor = "Danai Koutra and Claudia Plant and {Gomez Rodriguez}, Manuel and Elena Baralis and Francesco Bonchi",

booktitle = "Machine Learning and Knowledge Discovery in Databases",

address = "Germany",

}

Schultheis, E & Babbar, R 2023, Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. in D Koutra, C Plant, M Gomez Rodriguez, E Baralis & F Bonchi (eds), Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14171 LNAI, Springer, pp. 689-704, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Turin, Italy, 18/09/2023. https://doi.org/10.1007/978-3-031-43418-1_41

Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. / Schultheis, Erik; Babbar, Rohit.
Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. ed. / Danai Koutra; Claudia Plant; Manuel Gomez Rodriguez; Elena Baralis; Francesco Bonchi. Springer, 2023. p. 689-704 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14171 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU

AU - Schultheis, Erik

AU - Babbar, Rohit

PY - 2023

Y1 - 2023

N2 - In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied naïvely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse.

AB - In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied naïvely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse.

UR - http://www.scopus.com/inward/record.url?scp=85174438309&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-43418-1_41

DO - 10.1007/978-3-031-43418-1_41

M3 - Conference article in proceedings

AN - SCOPUS:85174438309

SN - 978-3-031-43417-4

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 689

EP - 704

BT - Machine Learning and Knowledge Discovery in Databases

A2 - Koutra, Danai

A2 - Plant, Claudia

A2 - Gomez Rodriguez, Manuel

A2 - Baralis, Elena

A2 - Bonchi, Francesco

PB - Springer

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases

Y2 - 18 September 2023 through 22 September 2023

ER -

Schultheis E, Babbar R. Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. In Koutra D, Plant C, Gomez Rodriguez M, Baralis E, Bonchi F, editors, Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. Springer. 2023. p. 689-704. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-43418-1_41

Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

ScaleX/Babbar: Scalable and Robust Representation Learning in Large output Spaces

HPC-HD/Babbar: High Performance Computing for the Detection and Analysis of Historical Discourses

Science-IT

Cite this

Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Projects

ScaleX/Babbar: Scalable and Robust Representation Learning in Large output Spaces

HPC-HD/Babbar: High Performance Computing for the Detection and Analysis of Historical Discourses

Equipment

Science-IT

Cite this