Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task

Nhan Phan; Anna von Zansen; Maria Kautonen; Ekaterina Voskoboinik; Tamas Grosz; Raili Hilden; Mikko Kurimo

doi:10.21437/interspeech.2024-1166

Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task

Nhan Phan^*, Anna von Zansen, Maria Kautonen, Ekaterina Voskoboinik, Tamas Grosz, Raili Hilden, Mikko Kurimo

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

3 Citations (Scopus)

12 Downloads (Pure)

Abstract

We propose a framework to address several unsolved challenges in second language (L2) automatic speaking assessment (ASA) and feedback. The challenges include: 1. ASA of visual task completion, 2. automated content grading and explanation of spontaneous L2 speech, 3. corrective feedback generation for L2 learners, and 4. all the above for a language that has minimal speech data of L2 learners. The proposed solution combines visual natural language generation (NLG), automatic speech recognition (ASR) and prompting a large language model (LLM) for low-resource L2 learners. We describe the solution and the outcomes of our case study for a picture description task in Finnish. Our results indicate substantial agreement with human experts in grading, explanation and feedback. This framework has the potential for a significant impact in constructing next-generation computer-assisted language learning systems to provide automatic scoring with feedback for learners of low-resource languages.

Original language	English
Title of host publication	Proceedings of the Interspeech 2024
Publisher	International Speech Communication Association (ISCA)
Pages	317-321
Number of pages	5
DOIs	https://doi.org/10.21437/interspeech.2024-1166
Publication status	Published - 1 Sept 2024
MoE publication type	A4 Conference publication
Event	Interspeech - Kos Island, Greece Duration: 1 Sept 2024 → 5 Sept 2024

Publication series

Name	Proceedings of the Annual Conference of the International Speech Communication Association
ISSN (Electronic)	2958-1796

Conference

Conference	Interspeech
Country/Territory	Greece
City	Kos Island
Period	01/09/2024 → 05/09/2024

Keywords

low-resource language
content feedback
Automatic Speech Assessment
L2 speaking
LLM

Access to Document

10.21437/interspeech.2024-1166

phan24_interspeechFinal published version, 946 KB

Cite this

Phan, N., von Zansen, A., Kautonen, M., Voskoboinik, E., Grosz, T., Hilden, R., & Kurimo, M. (2024). Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task. In Proceedings of the Interspeech 2024 (pp. 317-321). (Proceedings of the Annual Conference of the International Speech Communication Association). International Speech Communication Association (ISCA). https://doi.org/10.21437/interspeech.2024-1166

@inproceedings{5179e6161f9649769990a38dd606b944,

title = "Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task",

abstract = "We propose a framework to address several unsolved challenges in second language (L2) automatic speaking assessment (ASA) and feedback. The challenges include: 1. ASA of visual task completion, 2. automated content grading and explanation of spontaneous L2 speech, 3. corrective feedback generation for L2 learners, and 4. all the above for a language that has minimal speech data of L2 learners. The proposed solution combines visual natural language generation (NLG), automatic speech recognition (ASR) and prompting a large language model (LLM) for low-resource L2 learners. We describe the solution and the outcomes of our case study for a picture description task in Finnish. Our results indicate substantial agreement with human experts in grading, explanation and feedback. This framework has the potential for a significant impact in constructing next-generation computer-assisted language learning systems to provide automatic scoring with feedback for learners of low-resource languages.",

keywords = "low-resource language, content feedback, Automatic Speech Assessment, L2 speaking, LLM",

author = "Nhan Phan and \{von Zansen\}, Anna and Maria Kautonen and Ekaterina Voskoboinik and Tamas Grosz and Raili Hilden and Mikko Kurimo",

year = "2024",

month = sep,

day = "1",

doi = "10.21437/interspeech.2024-1166",

language = "English",

series = "Proceedings of the Annual Conference of the International Speech Communication Association",

publisher = "International Speech Communication Association (ISCA)",

pages = "317--321",

booktitle = "Proceedings of the Interspeech 2024",

address = "France",

note = "Interspeech ; Conference date: 01-09-2024 Through 05-09-2024",

}

Phan, N, von Zansen, A, Kautonen, M, Voskoboinik, E, Grosz, T, Hilden, R & Kurimo, M 2024, Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task. in Proceedings of the Interspeech 2024. Proceedings of the Annual Conference of the International Speech Communication Association, International Speech Communication Association (ISCA), pp. 317-321, Interspeech, Kos Island, Greece, 01/09/2024. https://doi.org/10.21437/interspeech.2024-1166

Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task. / Phan, Nhan; von Zansen, Anna; Kautonen, Maria et al.
Proceedings of the Interspeech 2024. International Speech Communication Association (ISCA), 2024. p. 317-321 (Proceedings of the Annual Conference of the International Speech Communication Association).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task

AU - Phan, Nhan

AU - von Zansen, Anna

AU - Kautonen, Maria

AU - Voskoboinik, Ekaterina

AU - Grosz, Tamas

AU - Hilden, Raili

AU - Kurimo, Mikko

PY - 2024/9/1

Y1 - 2024/9/1

N2 - We propose a framework to address several unsolved challenges in second language (L2) automatic speaking assessment (ASA) and feedback. The challenges include: 1. ASA of visual task completion, 2. automated content grading and explanation of spontaneous L2 speech, 3. corrective feedback generation for L2 learners, and 4. all the above for a language that has minimal speech data of L2 learners. The proposed solution combines visual natural language generation (NLG), automatic speech recognition (ASR) and prompting a large language model (LLM) for low-resource L2 learners. We describe the solution and the outcomes of our case study for a picture description task in Finnish. Our results indicate substantial agreement with human experts in grading, explanation and feedback. This framework has the potential for a significant impact in constructing next-generation computer-assisted language learning systems to provide automatic scoring with feedback for learners of low-resource languages.

AB - We propose a framework to address several unsolved challenges in second language (L2) automatic speaking assessment (ASA) and feedback. The challenges include: 1. ASA of visual task completion, 2. automated content grading and explanation of spontaneous L2 speech, 3. corrective feedback generation for L2 learners, and 4. all the above for a language that has minimal speech data of L2 learners. The proposed solution combines visual natural language generation (NLG), automatic speech recognition (ASR) and prompting a large language model (LLM) for low-resource L2 learners. We describe the solution and the outcomes of our case study for a picture description task in Finnish. Our results indicate substantial agreement with human experts in grading, explanation and feedback. This framework has the potential for a significant impact in constructing next-generation computer-assisted language learning systems to provide automatic scoring with feedback for learners of low-resource languages.

KW - low-resource language

KW - content feedback

KW - Automatic Speech Assessment

KW - L2 speaking

KW - LLM

UR - https://www.scopus.com/pages/publications/85214789913

U2 - 10.21437/interspeech.2024-1166

DO - 10.21437/interspeech.2024-1166

M3 - Conference article in proceedings

T3 - Proceedings of the Annual Conference of the International Speech Communication Association

SP - 317

EP - 321

BT - Proceedings of the Interspeech 2024

PB - International Speech Communication Association (ISCA)

T2 - Interspeech

Y2 - 1 September 2024 through 5 September 2024

ER -

Phan N, von Zansen A, Kautonen M, Voskoboinik E, Grosz T, Hilden R et al. Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task. In Proceedings of the Interspeech 2024. International Speech Communication Association (ISCA). 2024. p. 317-321. (Proceedings of the Annual Conference of the International Speech Communication Association). doi: 10.21437/interspeech.2024-1166

Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

AASIS: Automatic assessment of spoken interaction in second language

DigiTala: Aka-Digi Tala

Cite this

Automated content assessment and feedback for Finnish L2 learners in a picture description speaking task

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Projects

AASIS: Automatic assessment of spoken interaction in second language

DigiTala: Aka-Digi Tala

Cite this