Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0

Aku Rouhe; Anja Virkkunen; Juho Leinonen; Mikko Kurimo

Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0

Aku Rouhe^*, Anja Virkkunen, Juho Leinonen, Mikko Kurimo

^*Corresponding author for this work

Speech Recognition

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

51 Downloads (Pure)

Abstract

Low resource speech recognition can potentially benefit a lot from exploiting a pretrained model such as wav2vec 2.0. These pretrained models have learned useful representations in an unsupervised or self-supervised task, often leveraging a very large corpus of untranscribed speech. The pretrained models can then be used in various ways. In this work we compare two approaches which exploit wav2vec 2.0: an attention-based end-to-end model (AED), where the wav2vec 2.0 model is used in the model encoder, and a hybrid hidden Markov model (HMM/DNN) speech recognition system, where the wav2vec 2.0 model is used in the acoustic model. These approaches are compared in a very difficult Northern Sámi task, as well as an easier, simulated low resource task in Finnish. We find that the wav2vec 2.0 AED models can learn a working attention mechanism, but are still outperformed by wav2vec 2.0 HMM/DNN systems. Our best wav2vec 2.0 HMM/DNN recipe on 20 hours is competitive with an HMM/DNN system trained on 1600 hours.

Original language	English
Title of host publication	Proceedings of Interspeech'22
Publisher	International Speech Communication Association
Pages	3543-3547
Number of pages	5
Publication status	Published - 2022
MoE publication type	A4 Article in a conference publication
Event	Interspeech - Incheon, Korea, Republic of Duration: 18 Sep 2022 → 22 Sep 2022

Publication series

Name	Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
Publisher	International Speech Communication Association
ISSN (Print)	2308-457X
ISSN (Electronic)	1990-9772

Conference

Conference	Interspeech
Country/Territory	Korea, Republic of
City	Incheon
Period	18/09/2022 → 22/09/2022

Keywords

low resource
speech recognition
wav2vec 2.0

Access to Document

rouhe22_interspeechFinal published version, 179 KB

https://www.isca-speech.org/archive/pdfs/interspeech_2022/rouhe22_interspeech.pdf

SFX availability

Full text

Cite this

Rouhe, A., Virkkunen, A., Leinonen, J., & Kurimo, M. (2022). Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0. In Proceedings of Interspeech'22 (pp. 3543-3547). (Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH). International Speech Communication Association. https://www.isca-speech.org/archive/pdfs/interspeech_2022/rouhe22_interspeech.pdf

@inproceedings{abba134693044c6e99d984b0104d0e82,

title = "Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0",

abstract = "Low resource speech recognition can potentially benefit a lot from exploiting a pretrained model such as wav2vec 2.0. These pretrained models have learned useful representations in an unsupervised or self-supervised task, often leveraging a very large corpus of untranscribed speech. The pretrained models can then be used in various ways. In this work we compare two approaches which exploit wav2vec 2.0: an attention-based end-to-end model (AED), where the wav2vec 2.0 model is used in the model encoder, and a hybrid hidden Markov model (HMM/DNN) speech recognition system, where the wav2vec 2.0 model is used in the acoustic model. These approaches are compared in a very difficult Northern S{\'a}mi task, as well as an easier, simulated low resource task in Finnish. We find that the wav2vec 2.0 AED models can learn a working attention mechanism, but are still outperformed by wav2vec 2.0 HMM/DNN systems. Our best wav2vec 2.0 HMM/DNN recipe on 20 hours is competitive with an HMM/DNN system trained on 1600 hours.",

keywords = "low resource, speech recognition, wav2vec 2.0",

author = "Aku Rouhe and Anja Virkkunen and Juho Leinonen and Mikko Kurimo",

note = "Funding Information: We are grateful for the Academy of Finland project funding, numbers: 337073, 345790. We acknowledge the computational resources provided by the Aalto Science-IT project. Publisher Copyright: Copyright {\textcopyright} 2022 ISCA.; Interspeech ; Conference date: 18-09-2022 Through 22-09-2022",

year = "2022",

language = "English",

series = "Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH",

publisher = "International Speech Communication Association",

pages = "3543--3547",

booktitle = "Proceedings of Interspeech'22",

}

Rouhe, A , Virkkunen, A , Leinonen, J & Kurimo, M 2022, Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0. in Proceedings of Interspeech'22. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, International Speech Communication Association, pp. 3543-3547, Interspeech, Incheon, Korea, Republic of, 18/09/2022. <https://www.isca-speech.org/archive/pdfs/interspeech_2022/rouhe22_interspeech.pdf>

Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0. / Rouhe, Aku ; Virkkunen, Anja ; Leinonen, Juho et al.

Proceedings of Interspeech'22. International Speech Communication Association, 2022. p. 3543-3547 (Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0

AU - Rouhe, Aku

AU - Virkkunen, Anja

AU - Leinonen, Juho

AU - Kurimo, Mikko

N1 - Funding Information: We are grateful for the Academy of Finland project funding, numbers: 337073, 345790. We acknowledge the computational resources provided by the Aalto Science-IT project. Publisher Copyright: Copyright © 2022 ISCA.

PY - 2022

Y1 - 2022

N2 - Low resource speech recognition can potentially benefit a lot from exploiting a pretrained model such as wav2vec 2.0. These pretrained models have learned useful representations in an unsupervised or self-supervised task, often leveraging a very large corpus of untranscribed speech. The pretrained models can then be used in various ways. In this work we compare two approaches which exploit wav2vec 2.0: an attention-based end-to-end model (AED), where the wav2vec 2.0 model is used in the model encoder, and a hybrid hidden Markov model (HMM/DNN) speech recognition system, where the wav2vec 2.0 model is used in the acoustic model. These approaches are compared in a very difficult Northern Sámi task, as well as an easier, simulated low resource task in Finnish. We find that the wav2vec 2.0 AED models can learn a working attention mechanism, but are still outperformed by wav2vec 2.0 HMM/DNN systems. Our best wav2vec 2.0 HMM/DNN recipe on 20 hours is competitive with an HMM/DNN system trained on 1600 hours.

AB - Low resource speech recognition can potentially benefit a lot from exploiting a pretrained model such as wav2vec 2.0. These pretrained models have learned useful representations in an unsupervised or self-supervised task, often leveraging a very large corpus of untranscribed speech. The pretrained models can then be used in various ways. In this work we compare two approaches which exploit wav2vec 2.0: an attention-based end-to-end model (AED), where the wav2vec 2.0 model is used in the model encoder, and a hybrid hidden Markov model (HMM/DNN) speech recognition system, where the wav2vec 2.0 model is used in the acoustic model. These approaches are compared in a very difficult Northern Sámi task, as well as an easier, simulated low resource task in Finnish. We find that the wav2vec 2.0 AED models can learn a working attention mechanism, but are still outperformed by wav2vec 2.0 HMM/DNN systems. Our best wav2vec 2.0 HMM/DNN recipe on 20 hours is competitive with an HMM/DNN system trained on 1600 hours.

KW - low resource

KW - speech recognition

KW - wav2vec 2.0

UR - http://www.scopus.com/inward/record.url?scp=85140052172&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85140052172

T3 - Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

SP - 3543

EP - 3547

BT - Proceedings of Interspeech'22

PB - International Speech Communication Association

T2 - Interspeech

Y2 - 18 September 2022 through 22 September 2022

ER -

Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0

Abstract

Publication series

Conference

Keywords

Access to Document

SFX availability

Other files and links

Fingerprint

Understanding speech and scene with ears and eyes

Science-IT

Cite this

Low Resource Comparison of Attention-based and Hybrid ASR Exploiting wav2vec 2.0

Abstract

Publication series

Conference

Keywords

Access to Document

SFX availability

Other files and links

Fingerprint

Projects

Understanding speech and scene with ears and eyes

Equipment

Science-IT

Cite this