Projects per year
Abstract
Recently, BERT and Transformer-XL based architectures have achieved strong results in a range of NLP applications. In this paper, we explore Transformer architectures-BERT and Transformer-XL-as a language model for a Finnish ASR task with different rescoring schemes. We achieve strong results in both an intrinsic and an extrinsic task with Transformer-XL. Achieving 29% better perplexity and 3% better WER than our previous best LSTM-based approach. We also introduce a novel three-pass decoding scheme which improves the ASR performance by 8%. To the best of our knowledge, this is also the first work (i) to formulate an alpha smoothing framework to use the non-autoregressive BERT language model for an ASR task, and (ii) to explore sub-word units with Transformer-XL for an agglutinative language like Finnish.
Original language | English |
---|---|
Title of host publication | Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH |
Publisher | International Speech Communication Association |
Pages | 3630-3634 |
Number of pages | 5 |
Volume | 2020-October |
DOIs | |
Publication status | Published - 2020 |
MoE publication type | A4 Article in a conference publication |
Event | Interspeech - Shanghai, China Duration: 25 Oct 2020 → 29 Oct 2020 Conference number: 21 http://www.interspeech2020.org/ |
Publication series
Name | Interspeech |
---|---|
Publisher | International Speech Communication Association |
ISSN (Print) | 2308-457X |
Conference
Conference | Interspeech |
---|---|
Abbreviated title | INTERSPEECH |
Country/Territory | China |
City | Shanghai |
Period | 25/10/2020 → 29/10/2020 |
Internet address |
Keywords
- BERT
- Language modeling
- Speech recognition
- Transformer-XL
- Transformers
Fingerprint
Dive into the research topics of 'Finnish ASR with deep transformer models'. Together they form a unique fingerprint.-
Movie Making Finland: Finnish fiction films as audiovisual big data, 1907-2017
Kurimo, M., Moisio, A., Kathania, H., Porjazovski, D., Virkkunen, A. & Kathania, H.
01/01/2020 → 31/12/2022
Project: Academy of Finland: Other research funding
-
MeMAD: Methods for Managing Audiovisual Data: Combining Automatic Efficiency with Human Accuracy
Kurimo, M., Grósz, T., Raitio, R., Rouhe, A., Brander, T., Grönroos, S., Porjazovski, D. & Virkkunen, A.
27/12/2017 → 31/03/2021
Project: EU: Framework programmes funding