Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU

Erik Schultheis; Rohit Babbar

doi:10.1007/978-3-031-43418-1_41

Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU

^*Tämän työn vastaava kirjoittaja

University of Bath

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussa › Conference article in proceedings › Scientific › vertaisarvioitu

2 Sitaatiot (Scopus)

14 Lataukset (Pure)

Abstrakti

In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied naïvely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse.

Alkuperäiskieli	Englanti
Otsikko	Machine Learning and Knowledge Discovery in Databases
Alaotsikko	Research Track - European Conference, ECML PKDD 2023, Proceedings
Toimittajat	Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, Francesco Bonchi
Kustantaja	Springer
Sivut	689-704
Sivumäärä	16
ISBN (painettu)	978-3-031-43417-4
DOI - pysyväislinkit	https://doi.org/10.1007/978-3-031-43418-1_41
Tila	Julkaistu - 2023
OKM-julkaisutyyppi	A4 Artikkeli konferenssijulkaisussa
Tapahtuma	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - Turin, Italia Kesto: 18 syysk. 2023 → 22 syysk. 2023

Julkaisusarja

Nimi	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Kustantaja	Springer
Vuosikerta	14171 LNAI
ISSN (painettu)	0302-9743
ISSN (elektroninen)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases
Lyhennettä	ECML PKDD
Maa/Alue	Italia
Kaupunki	Turin
Ajanjakso	18/09/2023 → 22/09/2023

Pääsy asiakirjaan

10.1007/978-3-031-43418-1_41Lisenssi: CC BY

SCI_Schultheis_etal_ECML_PKDD_2023Final published version, 307 KBLisenssi: CC BY

Muut tiedostot ja linkit

Linkki julkaisuun Scopuksessa

1 Päättynyt
1 Aktiivinen

ScaleX/Babbar: Scalable and Robust Representation Learning in Large output Spaces
Babbar, R. (Vastuullinen tutkija)
01/09/2022 → 31/08/2026
Projekti: RCF Academy Project
HPC-HD/Babbar: High Performance Computing for the Detection and Analysis of Historical Discourses
Babbar, R. (Vastuullinen tutkija)
01/01/2022 → 31/12/2024
Projekti: RCF Academy Project targeted call

Science-IT
Hakala, M. (Manager)
Perustieteiden korkeakoulu
Laitteistot/tilat: Facility

Siteeraa tätä

Schultheis, E., & Babbar, R. (2023). Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. teoksessa D. Koutra, C. Plant, M. Gomez Rodriguez, E. Baralis, & F. Bonchi (Toimittajat), Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings (Sivut 689-704). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vuosikerta 14171 LNAI). Springer. https://doi.org/10.1007/978-3-031-43418-1_41

Schultheis, Erik ; Babbar, Rohit. / Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. Toimittaja / Danai Koutra ; Claudia Plant ; Manuel Gomez Rodriguez ; Elena Baralis ; Francesco Bonchi. Springer, 2023. Sivut 689-704 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{543dff7049444cddbe8a6df3365b6557,

title = "Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU",

abstract = "In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied na{\"i}vely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse.",

author = "Erik Schultheis and Rohit Babbar",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD ; Conference date: 18-09-2023 Through 22-09-2023",

year = "2023",

doi = "10.1007/978-3-031-43418-1_41",

language = "English",

isbn = "978-3-031-43417-4",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "689--704",

editor = "Danai Koutra and Claudia Plant and {Gomez Rodriguez}, Manuel and Elena Baralis and Francesco Bonchi",

booktitle = "Machine Learning and Knowledge Discovery in Databases",

address = "Germany",

}

Schultheis, E & Babbar, R 2023, Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. julkaisussa D Koutra, C Plant, M Gomez Rodriguez, E Baralis & F Bonchi (toim), Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vuosikerta 14171 LNAI, Springer, Sivut 689-704, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Turin, Italia, 18/09/2023. https://doi.org/10.1007/978-3-031-43418-1_41

Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. / Schultheis, Erik; Babbar, Rohit.
Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. toim. / Danai Koutra; Claudia Plant; Manuel Gomez Rodriguez; Elena Baralis; Francesco Bonchi. Springer, 2023. s. 689-704 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vuosikerta 14171 LNAI).

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussa › Conference article in proceedings › Scientific › vertaisarvioitu

TY - GEN

T1 - Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU

AU - Schultheis, Erik

AU - Babbar, Rohit

PY - 2023

Y1 - 2023

N2 - In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied naïvely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse.

AB - In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied naïvely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse.

UR - http://www.scopus.com/inward/record.url?scp=85174438309&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-43418-1_41

DO - 10.1007/978-3-031-43418-1_41

M3 - Conference article in proceedings

AN - SCOPUS:85174438309

SN - 978-3-031-43417-4

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 689

EP - 704

BT - Machine Learning and Knowledge Discovery in Databases

A2 - Koutra, Danai

A2 - Plant, Claudia

A2 - Gomez Rodriguez, Manuel

A2 - Baralis, Elena

A2 - Bonchi, Francesco

PB - Springer

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases

Y2 - 18 September 2023 through 22 September 2023

ER -

Schultheis E, Babbar R. Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU. julkaisussa Koutra D, Plant C, Gomez Rodriguez M, Baralis E, Bonchi F, toimittajat, Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Proceedings. Springer. 2023. s. 689-704. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-43418-1_41

Towards Memory-Efficient Training for Extremely Large Output Spaces : Learning with 670k Labels on a Single Commodity GPU

Abstrakti

Julkaisusarja

Conference

Pääsy asiakirjaan

Muut tiedostot ja linkit

Sormenjälki

Projektit

ScaleX/Babbar: Scalable and Robust Representation Learning in Large output Spaces

HPC-HD/Babbar: High Performance Computing for the Detection and Analysis of Historical Discourses

Laitteet

Science-IT

Siteeraa tätä