Multi-node Training for StyleGAN2

Niki Loppi; Tuomas Kynkäänniemi

doi:10.1007/978-3-030-68763-2_51

Multi-node Training for StyleGAN2

NVIDIA AI Technology Center

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

260 Downloads (Pure)

Abstract

StyleGAN2 is a Tensorflow-based Generative Adversarial Network (GAN) framework that represents the state-of-the-art in generative image modelling. The current release of StyleGAN2 implements multi-GPU training via Tensorflow’s device contexts which limits data parallelism to a single node. In this work, a data-parallel multi-node training capability is implemented in StyleGAN2 via Horovod which enables harnessing the compute capability of larger cluster architectures. We demonstrate that the new Horovod-based communication outperforms the previous context approach on a single node. Furthermore, we demonstrate that the multi-node training does not compromise the accuracy of StyleGAN2 for a constant effective batch size. Finally, we report strong and weak scaling of the new implementation up to 64 NVIDIA Tesla A100 GPUs distributed across eight NVIDIA DGX A100 nodes, demonstrating the utility of the approach at scale.

Original language	English
Title of host publication	Pattern Recognition. ICPR International Workshops and Challenges, 2021, Proceedings
Editors	Alberto Del Bimbo, Rita Cucchiara, Stan Sclaroff, Giovanni Maria Farinella, Tao Mei, Marco Bertini, Hugo Jair Escalante, Roberto Vezzani
Publisher	Springer
Pages	677-684
Number of pages	8
ISBN (Print)	978-3-030-68762-5
DOIs	https://doi.org/10.1007/978-3-030-68763-2_51
Publication status	Published - 2021
MoE publication type	A4 Conference publication
Event	International Conference on Pattern Recognition - Virtual, Online, Milan, Italy Duration: 10 Jan 2021 → 15 Jan 2021 Conference number: 25

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Publisher	Springer
Volume	12661 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	International Conference on Pattern Recognition
Abbreviated title	ICPR
Country/Territory	Italy
City	Milan
Period	10/01/2021 → 15/01/2021

Keywords

GAN
GPU
Massively parallel architectures
Multi-node training
StyleGAN2

Access to Document

10.1007/978-3-030-68763-2_51

SCI_Loppi_Multinode_StyleGAN2_icpr2020_workshopAccepted author manuscript, 703 KB

Cite this

Loppi, N., & Kynkäänniemi, T. (2021). Multi-node Training for StyleGAN2. In A. Del Bimbo, R. Cucchiara, S. Sclaroff, G. M. Farinella, T. Mei, M. Bertini, H. J. Escalante, & R. Vezzani (Eds.), Pattern Recognition. ICPR International Workshops and Challenges, 2021, Proceedings (pp. 677-684). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12661 LNCS). Springer. https://doi.org/10.1007/978-3-030-68763-2_51

Loppi, Niki ; Kynkäänniemi, Tuomas. / Multi-node Training for StyleGAN2. Pattern Recognition. ICPR International Workshops and Challenges, 2021, Proceedings. editor / Alberto Del Bimbo ; Rita Cucchiara ; Stan Sclaroff ; Giovanni Maria Farinella ; Tao Mei ; Marco Bertini ; Hugo Jair Escalante ; Roberto Vezzani. Springer, 2021. pp. 677-684 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{80a4a8646413418a96f87ca0a189b1e4,

title = "Multi-node Training for StyleGAN2",

abstract = "StyleGAN2 is a Tensorflow-based Generative Adversarial Network (GAN) framework that represents the state-of-the-art in generative image modelling. The current release of StyleGAN2 implements multi-GPU training via Tensorflow{\textquoteright}s device contexts which limits data parallelism to a single node. In this work, a data-parallel multi-node training capability is implemented in StyleGAN2 via Horovod which enables harnessing the compute capability of larger cluster architectures. We demonstrate that the new Horovod-based communication outperforms the previous context approach on a single node. Furthermore, we demonstrate that the multi-node training does not compromise the accuracy of StyleGAN2 for a constant effective batch size. Finally, we report strong and weak scaling of the new implementation up to 64 NVIDIA Tesla A100 GPUs distributed across eight NVIDIA DGX A100 nodes, demonstrating the utility of the approach at scale.",

keywords = "GAN, GPU, Massively parallel architectures, Multi-node training, StyleGAN2",

author = "Niki Loppi and Tuomas Kynk{\"a}{\"a}nniemi",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.; International Conference on Pattern Recognition, ICPR ; Conference date: 10-01-2021 Through 15-01-2021",

year = "2021",

doi = "10.1007/978-3-030-68763-2_51",

language = "English",

isbn = "978-3-030-68762-5",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "677--684",

editor = "{Del Bimbo}, Alberto and Rita Cucchiara and Stan Sclaroff and Farinella, {Giovanni Maria} and Tao Mei and Marco Bertini and Escalante, {Hugo Jair} and Roberto Vezzani",

booktitle = "Pattern Recognition. ICPR International Workshops and Challenges, 2021, Proceedings",

address = "Germany",

}

Loppi, N & Kynkäänniemi, T 2021, Multi-node Training for StyleGAN2. in A Del Bimbo, R Cucchiara, S Sclaroff, GM Farinella, T Mei, M Bertini, HJ Escalante & R Vezzani (eds), Pattern Recognition. ICPR International Workshops and Challenges, 2021, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12661 LNCS, Springer, pp. 677-684, International Conference on Pattern Recognition, Milan, Italy, 10/01/2021. https://doi.org/10.1007/978-3-030-68763-2_51

Multi-node Training for StyleGAN2. / Loppi, Niki; Kynkäänniemi, Tuomas.
Pattern Recognition. ICPR International Workshops and Challenges, 2021, Proceedings. ed. / Alberto Del Bimbo; Rita Cucchiara; Stan Sclaroff; Giovanni Maria Farinella; Tao Mei; Marco Bertini; Hugo Jair Escalante; Roberto Vezzani. Springer, 2021. p. 677-684 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12661 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - Multi-node Training for StyleGAN2

AU - Loppi, Niki

AU - Kynkäänniemi, Tuomas

N1 - Conference code: 25

PY - 2021

Y1 - 2021

N2 - StyleGAN2 is a Tensorflow-based Generative Adversarial Network (GAN) framework that represents the state-of-the-art in generative image modelling. The current release of StyleGAN2 implements multi-GPU training via Tensorflow’s device contexts which limits data parallelism to a single node. In this work, a data-parallel multi-node training capability is implemented in StyleGAN2 via Horovod which enables harnessing the compute capability of larger cluster architectures. We demonstrate that the new Horovod-based communication outperforms the previous context approach on a single node. Furthermore, we demonstrate that the multi-node training does not compromise the accuracy of StyleGAN2 for a constant effective batch size. Finally, we report strong and weak scaling of the new implementation up to 64 NVIDIA Tesla A100 GPUs distributed across eight NVIDIA DGX A100 nodes, demonstrating the utility of the approach at scale.

AB - StyleGAN2 is a Tensorflow-based Generative Adversarial Network (GAN) framework that represents the state-of-the-art in generative image modelling. The current release of StyleGAN2 implements multi-GPU training via Tensorflow’s device contexts which limits data parallelism to a single node. In this work, a data-parallel multi-node training capability is implemented in StyleGAN2 via Horovod which enables harnessing the compute capability of larger cluster architectures. We demonstrate that the new Horovod-based communication outperforms the previous context approach on a single node. Furthermore, we demonstrate that the multi-node training does not compromise the accuracy of StyleGAN2 for a constant effective batch size. Finally, we report strong and weak scaling of the new implementation up to 64 NVIDIA Tesla A100 GPUs distributed across eight NVIDIA DGX A100 nodes, demonstrating the utility of the approach at scale.

KW - GAN

KW - GPU

KW - Massively parallel architectures

KW - Multi-node training

KW - StyleGAN2

UR - http://www.scopus.com/inward/record.url?scp=85104320857&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-68763-2_51

DO - 10.1007/978-3-030-68763-2_51

M3 - Conference article in proceedings

AN - SCOPUS:85104320857

SN - 978-3-030-68762-5

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 677

EP - 684

BT - Pattern Recognition. ICPR International Workshops and Challenges, 2021, Proceedings

A2 - Del Bimbo, Alberto

A2 - Cucchiara, Rita

A2 - Sclaroff, Stan

A2 - Farinella, Giovanni Maria

A2 - Mei, Tao

A2 - Bertini, Marco

A2 - Escalante, Hugo Jair

A2 - Vezzani, Roberto

PB - Springer

T2 - International Conference on Pattern Recognition

Y2 - 10 January 2021 through 15 January 2021

ER -

Loppi N, Kynkäänniemi T. Multi-node Training for StyleGAN2. In Del Bimbo A, Cucchiara R, Sclaroff S, Farinella GM, Mei T, Bertini M, Escalante HJ, Vezzani R, editors, Pattern Recognition. ICPR International Workshops and Challenges, 2021, Proceedings. Springer. 2021. p. 677-684. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-68763-2_51

Multi-node Training for StyleGAN2

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this