Wav2vec2-based Paralinguistic Systems to Recognise Vocalised Emotions and Stuttering

Tamás Grósz; Dejan Porjazovski; Yaroslav Getman; Sudarsana Kadiri; Mikko Kurimo

doi:10.1145/3503161.3551572

Wav2vec2-based Paralinguistic Systems to Recognise Vocalised Emotions and Stuttering

Tamás Grósz, Dejan Porjazovski, Yaroslav Getman, Sudarsana Kadiri, Mikko Kurimo

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

131 Downloads (Pure)

Abstract

With the rapid advancement in automatic speech recognition and
natural language understanding, a complementary field (paralin-
guistics) emerged, focusing on the non-verbal content of speech.
The ACM Multimedia 2022 Computational Paralinguistics Challenge introduced several exciting tasks of this field. In this work, we
focus on tackling two Sub-Challenges using modern, pre-trained
models called wav2vec2. Our experimental results demonstrated
that wav2vec2 is an excellent tool for detecting the emotions behind vocalisations and recognising different types of stutterings.
Albeit they achieve outstanding results on their own, our results
demonstrated that wav2vec2-based systems could be further improved by ensembling them with other models. Our best systems
outperformed the competition baselines by a considerable margin,
achieving an unweighted average recall of 44.0 (absolute improvement of 6.6% over baseline) on the Vocalisation Sub-Challenge and
62.1 (absolute improvement of 21.7% over baseline) on the Stuttering
Sub-Challenge.

Original language	English
Title of host publication	Proceedings of the 30th ACM International Conference on Multimedia
Publisher	ACM
Number of pages	4
ISBN (Electronic)	978-1-4503-9203-7
DOIs	https://doi.org/10.1145/3503161.3551572
Publication status	Published - Oct 2022
MoE publication type	A4 Conference publication
Event	ACM International Conference on Multimedia - Lisboa, Portugal Duration: 10 Oct 2022 → 14 Oct 2022 Conference number: 30

Conference

Conference	ACM International Conference on Multimedia
Abbreviated title	MM
Country/Territory	Portugal
City	Lisboa
Period	10/10/2022 → 14/10/2022

Access to Document

10.1145/3503161.3551572Licence: CC BY

Wav2Vec2_StutteringFinal published version, 892 KBLicence: CC BY

USSEE: Understanding Speech and Scene with Ears and Eyes
Kurimo, M. (Principal investigator), Virkkunen, A. (Project Member) & Grósz, T. (Project Member)
01/01/2022 → 31/12/2024
Project: Academy of Finland: Other research funding
TEFLON: Technology-enhanced foreign and second-language learning of Nordic languages
Kurimo, M. (Principal investigator), Al-Ghezi, R. (Project Member), Smolander, A.-R. (Project Member), Getman, Y. (Project Member), Phan, N. (Project Member), Grósz, T. (Project Member) & Voskoboinik, E. (Project Member)
01/04/2021 → 31/03/2025
Project: Other external funding: Other foreign funding

Science-IT
Hakala, M. (Manager)
School of Science
Facility/equipment: Facility

Cite this

@inproceedings{72c5bd61bd3e43b3affec06ee805500b,

title = "Wav2vec2-based Paralinguistic Systems to Recognise Vocalised Emotions and Stuttering",

abstract = "With the rapid advancement in automatic speech recognition andnatural language understanding, a complementary field (paralin-guistics) emerged, focusing on the non-verbal content of speech.The ACM Multimedia 2022 Computational Paralinguistics Challenge introduced several exciting tasks of this field. In this work, wefocus on tackling two Sub-Challenges using modern, pre-trainedmodels called wav2vec2. Our experimental results demonstratedthat wav2vec2 is an excellent tool for detecting the emotions behind vocalisations and recognising different types of stutterings.Albeit they achieve outstanding results on their own, our resultsdemonstrated that wav2vec2-based systems could be further improved by ensembling them with other models. Our best systemsoutperformed the competition baselines by a considerable margin,achieving an unweighted average recall of 44.0 (absolute improvement of 6.6% over baseline) on the Vocalisation Sub-Challenge and62.1 (absolute improvement of 21.7% over baseline) on the StutteringSub-Challenge.",

author = "Tam{\'a}s Gr{\'o}sz and Dejan Porjazovski and Yaroslav Getman and Sudarsana Kadiri and Mikko Kurimo",

year = "2022",

month = oct,

doi = "10.1145/3503161.3551572",

language = "English",

booktitle = "Proceedings of the 30th ACM International Conference on Multimedia",

publisher = "ACM",

address = "United States",

note = "ACM International Conference on Multimedia, MM ; Conference date: 10-10-2022 Through 14-10-2022",

}

TY - GEN

T1 - Wav2vec2-based Paralinguistic Systems to Recognise Vocalised Emotions and Stuttering

AU - Grósz, Tamás

AU - Porjazovski, Dejan

AU - Getman, Yaroslav

AU - Kadiri, Sudarsana

AU - Kurimo, Mikko

N1 - Conference code: 30

PY - 2022/10

Y1 - 2022/10

N2 - With the rapid advancement in automatic speech recognition andnatural language understanding, a complementary field (paralin-guistics) emerged, focusing on the non-verbal content of speech.The ACM Multimedia 2022 Computational Paralinguistics Challenge introduced several exciting tasks of this field. In this work, wefocus on tackling two Sub-Challenges using modern, pre-trainedmodels called wav2vec2. Our experimental results demonstratedthat wav2vec2 is an excellent tool for detecting the emotions behind vocalisations and recognising different types of stutterings.Albeit they achieve outstanding results on their own, our resultsdemonstrated that wav2vec2-based systems could be further improved by ensembling them with other models. Our best systemsoutperformed the competition baselines by a considerable margin,achieving an unweighted average recall of 44.0 (absolute improvement of 6.6% over baseline) on the Vocalisation Sub-Challenge and62.1 (absolute improvement of 21.7% over baseline) on the StutteringSub-Challenge.

AB - With the rapid advancement in automatic speech recognition andnatural language understanding, a complementary field (paralin-guistics) emerged, focusing on the non-verbal content of speech.The ACM Multimedia 2022 Computational Paralinguistics Challenge introduced several exciting tasks of this field. In this work, wefocus on tackling two Sub-Challenges using modern, pre-trainedmodels called wav2vec2. Our experimental results demonstratedthat wav2vec2 is an excellent tool for detecting the emotions behind vocalisations and recognising different types of stutterings.Albeit they achieve outstanding results on their own, our resultsdemonstrated that wav2vec2-based systems could be further improved by ensembling them with other models. Our best systemsoutperformed the competition baselines by a considerable margin,achieving an unweighted average recall of 44.0 (absolute improvement of 6.6% over baseline) on the Vocalisation Sub-Challenge and62.1 (absolute improvement of 21.7% over baseline) on the StutteringSub-Challenge.

U2 - 10.1145/3503161.3551572

DO - 10.1145/3503161.3551572

M3 - Conference article in proceedings

BT - Proceedings of the 30th ACM International Conference on Multimedia

PB - ACM

T2 - ACM International Conference on Multimedia

Y2 - 10 October 2022 through 14 October 2022

ER -

Wav2vec2-based Paralinguistic Systems to Recognise Vocalised Emotions and Stuttering

Abstract

Conference

Access to Document

Fingerprint

Projects

USSEE: Understanding Speech and Scene with Ears and Eyes

TEFLON: Technology-enhanced foreign and second-language learning of Nordic languages

Equipment

Science-IT

Cite this