Finnish ASR with deep transformer models

Abhilash Jain; Aku Rouhe; Stig Arne Grönroos; Mikko Kurimo

doi:10.21437/Interspeech.2020-1784

Finnish ASR with deep transformer models

Abhilash Jain, Aku Rouhe, Stig Arne Grönroos, Mikko Kurimo

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

45 Downloads (Pure)

Abstract

Recently, BERT and Transformer-XL based architectures have achieved strong results in a range of NLP applications. In this paper, we explore Transformer architectures-BERT and Transformer-XL-as a language model for a Finnish ASR task with different rescoring schemes. We achieve strong results in both an intrinsic and an extrinsic task with Transformer-XL. Achieving 29% better perplexity and 3% better WER than our previous best LSTM-based approach. We also introduce a novel three-pass decoding scheme which improves the ASR performance by 8%. To the best of our knowledge, this is also the first work (i) to formulate an alpha smoothing framework to use the non-autoregressive BERT language model for an ASR task, and (ii) to explore sub-word units with Transformer-XL for an agglutinative language like Finnish.

Original language	English
Title of host publication	Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
Publisher	International Speech Communication Association
Pages	3630-3634
Number of pages	5
Volume	2020-October
DOIs	https://doi.org/10.21437/Interspeech.2020-1784
Publication status	Published - 2020
MoE publication type	A4 Article in a conference publication
Event	Interspeech - Shanghai, China Duration: 25 Oct 2020 → 29 Oct 2020 Conference number: 21 http://www.interspeech2020.org/

Publication series

Name	Interspeech
Publisher	International Speech Communication Association
ISSN (Print)	2308-457X

Conference

Conference	Interspeech
Abbreviated title	INTERSPEECH
Country	China
City	Shanghai
Period	25/10/2020 → 29/10/2020
Internet address	http://www.interspeech2020.org/

Keywords

BERT
Language modeling
Speech recognition
Transformer-XL
Transformers

Access to Document

10.21437/Interspeech.2020-1784

Finnish ASR with Deep Transformer ModelsFinal published version, 193 KB

Link to publication in Scopus

Fingerprint Dive into the research topics of 'Finnish ASR with deep transformer models'. Together they form a unique fingerprint.

2 Active

Movie Making Finland: Finnish fiction films as audiovisual big data, 1907-2017
Kurimo, M. & Kathania, H.
01/01/2020 → 31/12/2022
Project: Academy of Finland: Other research funding
MeMAD: Methods for Managing Audiovisual Data: Combining Automatic Efficiency with Human Accuracy
Raitio, R., Kurimo, M., Grönroos, S., Brander, T., Rouhe, A., Grósz, T., Virkkunen, A. & Porjazovski, D.
27/12/2017 → 31/03/2021
Project: EU: Framework programmes funding

Science-IT
Mikko Hakala (Manager)
School of Science
Facility/equipment: Facility

Cite this

@inproceedings{d9c019cef31a42f49cb8e519b2b39c32,

title = "Finnish ASR with deep transformer models",

abstract = "Recently, BERT and Transformer-XL based architectures have achieved strong results in a range of NLP applications. In this paper, we explore Transformer architectures-BERT and Transformer-XL-as a language model for a Finnish ASR task with different rescoring schemes. We achieve strong results in both an intrinsic and an extrinsic task with Transformer-XL. Achieving 29% better perplexity and 3% better WER than our previous best LSTM-based approach. We also introduce a novel three-pass decoding scheme which improves the ASR performance by 8%. To the best of our knowledge, this is also the first work (i) to formulate an alpha smoothing framework to use the non-autoregressive BERT language model for an ASR task, and (ii) to explore sub-word units with Transformer-XL for an agglutinative language like Finnish.",

keywords = "BERT, Language modeling, Speech recognition, Transformer-XL, Transformers",

author = "Abhilash Jain and Aku Rouhe and Gr{\"o}nroos, {Stig Arne} and Mikko Kurimo",

note = "| openaire: EC/H2020/780069/EU//MeMAD; Interspeech, INTERSPEECH ; Conference date: 25-10-2020 Through 29-10-2020",

year = "2020",

doi = "10.21437/Interspeech.2020-1784",

language = "English",

volume = "2020-October",

series = "Interspeech",

publisher = "International Speech Communication Association",

pages = "3630--3634",

booktitle = "Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH",

url = "http://www.interspeech2020.org/",

}

Jain, A, Rouhe, A , Grönroos, SA & Kurimo, M 2020, Finnish ASR with deep transformer models. in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. vol. 2020-October, Interspeech, International Speech Communication Association, pp. 3630-3634, Interspeech, Shanghai, China, 25/10/2020. https://doi.org/10.21437/Interspeech.2020-1784

Finnish ASR with deep transformer models. / Jain, Abhilash; Rouhe, Aku ; Grönroos, Stig Arne ; Kurimo, Mikko.

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. Vol. 2020-October International Speech Communication Association, 2020. p. 3630-3634 (Interspeech).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Finnish ASR with deep transformer models

AU - Jain, Abhilash

AU - Rouhe, Aku

AU - Grönroos, Stig Arne

AU - Kurimo, Mikko

N1 - Conference code: 21

PY - 2020

Y1 - 2020

N2 - Recently, BERT and Transformer-XL based architectures have achieved strong results in a range of NLP applications. In this paper, we explore Transformer architectures-BERT and Transformer-XL-as a language model for a Finnish ASR task with different rescoring schemes. We achieve strong results in both an intrinsic and an extrinsic task with Transformer-XL. Achieving 29% better perplexity and 3% better WER than our previous best LSTM-based approach. We also introduce a novel three-pass decoding scheme which improves the ASR performance by 8%. To the best of our knowledge, this is also the first work (i) to formulate an alpha smoothing framework to use the non-autoregressive BERT language model for an ASR task, and (ii) to explore sub-word units with Transformer-XL for an agglutinative language like Finnish.

AB - Recently, BERT and Transformer-XL based architectures have achieved strong results in a range of NLP applications. In this paper, we explore Transformer architectures-BERT and Transformer-XL-as a language model for a Finnish ASR task with different rescoring schemes. We achieve strong results in both an intrinsic and an extrinsic task with Transformer-XL. Achieving 29% better perplexity and 3% better WER than our previous best LSTM-based approach. We also introduce a novel three-pass decoding scheme which improves the ASR performance by 8%. To the best of our knowledge, this is also the first work (i) to formulate an alpha smoothing framework to use the non-autoregressive BERT language model for an ASR task, and (ii) to explore sub-word units with Transformer-XL for an agglutinative language like Finnish.

KW - BERT

KW - Language modeling

KW - Speech recognition

KW - Transformer-XL

KW - Transformers

UR - http://www.scopus.com/inward/record.url?scp=85098184485&partnerID=8YFLogxK

U2 - 10.21437/Interspeech.2020-1784

DO - 10.21437/Interspeech.2020-1784

M3 - Conference contribution

AN - SCOPUS:85098184485

VL - 2020-October

T3 - Interspeech

SP - 3630

EP - 3634

BT - Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

PB - International Speech Communication Association

T2 - Interspeech

Y2 - 25 October 2020 through 29 October 2020

ER -

Finnish ASR with deep transformer models

Abstract

Publication series

Conference

Keywords

Access to Document

Movie Making Finland: Finnish fiction films as audiovisual big data, 1907-2017

MeMAD: Methods for Managing Audiovisual Data: Combining Automatic Efficiency with Human Accuracy

Science-IT

Cite this