Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings

Aku Rouhe; Tuomas Kaseva; Mikko Kurimo

doi:10.1109/ICASSP40776.2020.9053998

Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings

Aku Rouhe, Tuomas Kaseva, Mikko Kurimo

Speech Recognition

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

3 Citations (Scopus)

132 Downloads (Pure)

Abstract

In speaker-aware training, a speaker embedding is appended to DNN input features. This allows the DNN to effectively learn representations, which are robust to speaker variability.We apply speaker-aware training to attention-based end-to-end speech recognition. We show that it can improve over a purely end-to-end baseline. We also propose speaker-aware training as a viable method to leverage untranscribed, speaker annotated data.We apply state-of-the-art embedding approaches, both i-vectors and neural embeddings, such as x-vectors. We experiment with embeddings trained in two conditions: on the fixed ASR data, and on a large untranscribed dataset. We run our experiments on the TED-LIUM and Wall Street Journal datasets. No embedding consistently outperforms all others, but in many settings neural embeddings outperform i-vectors.

Original language	English
Title of host publication	2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings
Publisher	IEEE
Pages	7064-7068
Number of pages	5
ISBN (Electronic)	9781509066315
DOIs	https://doi.org/10.1109/ICASSP40776.2020.9053998
Publication status	Published - May 2020
MoE publication type	A4 Article in a conference publication
Event	IEEE International Conference on Acoustics, Speech, and Signal Processing - Virtual conference, Barcelona, Spain Duration: 4 May 2020 → 8 May 2020 Conference number: 45

Publication series

Name	Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing
ISSN (Print)	1520-6149
ISSN (Electronic)	2379-190X

Conference

Conference	IEEE International Conference on Acoustics, Speech, and Signal Processing
Abbreviated title	ICASSP
Country/Territory	Spain
City	Barcelona
Period	04/05/2020 → 08/05/2020
Other	Virtual conference

Keywords

end-to-end speech recognition
speaker embedding
speaker-adaptation
speaker-aware training

Access to Document

10.1109/ICASSP40776.2020.9053998

Rouhe_Speaker-aware_training_of_attention_based_end-to-end_speech_recognitionAccepted author manuscript, 111 KB

OpenUrl availability

Full text

Cite this

Rouhe, A., Kaseva, T., & Kurimo, M. (2020). Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings. In 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings (pp. 7064-7068). [9053998] (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing). IEEE. https://doi.org/10.1109/ICASSP40776.2020.9053998

Rouhe, Aku ; Kaseva, Tuomas ; Kurimo, Mikko. / Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings. 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings. IEEE, 2020. pp. 7064-7068 (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing).

@inproceedings{17b37434aaee43d89718888409b279c4,

title = "Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings",

abstract = "In speaker-aware training, a speaker embedding is appended to DNN input features. This allows the DNN to effectively learn representations, which are robust to speaker variability.We apply speaker-aware training to attention-based end-to-end speech recognition. We show that it can improve over a purely end-to-end baseline. We also propose speaker-aware training as a viable method to leverage untranscribed, speaker annotated data.We apply state-of-the-art embedding approaches, both i-vectors and neural embeddings, such as x-vectors. We experiment with embeddings trained in two conditions: on the fixed ASR data, and on a large untranscribed dataset. We run our experiments on the TED-LIUM and Wall Street Journal datasets. No embedding consistently outperforms all others, but in many settings neural embeddings outperform i-vectors.",

keywords = "end-to-end speech recognition, speaker embedding, speaker-adaptation, speaker-aware training",

author = "Aku Rouhe and Tuomas Kaseva and Mikko Kurimo",

note = "| openaire: EC/H2020/780069/EU//MeMAD; IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP ; Conference date: 04-05-2020 Through 08-05-2020",

year = "2020",

month = may,

doi = "10.1109/ICASSP40776.2020.9053998",

language = "English",

series = "Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing",

publisher = "IEEE",

pages = "7064--7068",

booktitle = "2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings",

address = "United States",

}

Rouhe, A, Kaseva, T & Kurimo, M 2020, Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings. in 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings., 9053998, Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, IEEE, pp. 7064-7068, IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain, 04/05/2020. https://doi.org/10.1109/ICASSP40776.2020.9053998

Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings. / Rouhe, Aku; Kaseva, Tuomas; Kurimo, Mikko.

2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings. IEEE, 2020. p. 7064-7068 9053998 (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings

AU - Rouhe, Aku

AU - Kaseva, Tuomas

AU - Kurimo, Mikko

N1 - Conference code: 45

PY - 2020/5

Y1 - 2020/5

N2 - In speaker-aware training, a speaker embedding is appended to DNN input features. This allows the DNN to effectively learn representations, which are robust to speaker variability.We apply speaker-aware training to attention-based end-to-end speech recognition. We show that it can improve over a purely end-to-end baseline. We also propose speaker-aware training as a viable method to leverage untranscribed, speaker annotated data.We apply state-of-the-art embedding approaches, both i-vectors and neural embeddings, such as x-vectors. We experiment with embeddings trained in two conditions: on the fixed ASR data, and on a large untranscribed dataset. We run our experiments on the TED-LIUM and Wall Street Journal datasets. No embedding consistently outperforms all others, but in many settings neural embeddings outperform i-vectors.

AB - In speaker-aware training, a speaker embedding is appended to DNN input features. This allows the DNN to effectively learn representations, which are robust to speaker variability.We apply speaker-aware training to attention-based end-to-end speech recognition. We show that it can improve over a purely end-to-end baseline. We also propose speaker-aware training as a viable method to leverage untranscribed, speaker annotated data.We apply state-of-the-art embedding approaches, both i-vectors and neural embeddings, such as x-vectors. We experiment with embeddings trained in two conditions: on the fixed ASR data, and on a large untranscribed dataset. We run our experiments on the TED-LIUM and Wall Street Journal datasets. No embedding consistently outperforms all others, but in many settings neural embeddings outperform i-vectors.

KW - end-to-end speech recognition

KW - speaker embedding

KW - speaker-adaptation

KW - speaker-aware training

UR - http://www.scopus.com/inward/record.url?scp=85089246234&partnerID=8YFLogxK

U2 - 10.1109/ICASSP40776.2020.9053998

DO - 10.1109/ICASSP40776.2020.9053998

M3 - Conference contribution

AN - SCOPUS:85089246234

T3 - Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing

SP - 7064

EP - 7068

BT - 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings

PB - IEEE

T2 - IEEE International Conference on Acoustics, Speech, and Signal Processing

Y2 - 4 May 2020 through 8 May 2020

ER -

Rouhe A, Kaseva T, Kurimo M. Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings. In 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings. IEEE. 2020. p. 7064-7068. 9053998. (Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing). doi: 10.1109/ICASSP40776.2020.9053998

Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings

Abstract

Publication series

Conference

Keywords

Access to Document

OpenUrl availability

Other files and links

Fingerprint

MeMAD

Science-IT

Cite this

Speaker-Aware Training of Attention-Based End-to-End Speech Recognition Using Neural Speaker Embeddings

Abstract

Publication series

Conference

Keywords

Access to Document

OpenUrl availability

Other files and links

Fingerprint

Projects

MeMAD

Equipment

Science-IT

Cite this