Projects per year
Abstract
We introduce the CaptainA mobile app, designed to meet
the needs of second language (L2) learners engaged in self-
study of Finnish, with potential applicability to other languages.
Our app can provide automatic speaking assessment (ASA) of
task completion in picture-based tasks, along with grading ex-
planations and corrective feedback. It can also automatically
generate pictures for visual tasks, providing users with unlim-
ited practice opportunities. The mobile app is based on our
framework that combines visual natural language generation
(NLG), automatic speech recognition (ASR), and prompting
large language model (LLM) for low-resource language. Our
goal is to promote the development of next-generation speech-
based computer-assisted language learning (CALL) systems ca-
pable of providing automatic scoring with feedback for learn-
ers, even when minimal speech data of L2 learners is available.
While the mobile app demonstration is designed for Finnish, the
app can also be tested in English.
the needs of second language (L2) learners engaged in self-
study of Finnish, with potential applicability to other languages.
Our app can provide automatic speaking assessment (ASA) of
task completion in picture-based tasks, along with grading ex-
planations and corrective feedback. It can also automatically
generate pictures for visual tasks, providing users with unlim-
ited practice opportunities. The mobile app is based on our
framework that combines visual natural language generation
(NLG), automatic speech recognition (ASR), and prompting
large language model (LLM) for low-resource language. Our
goal is to promote the development of next-generation speech-
based computer-assisted language learning (CALL) systems ca-
pable of providing automatic scoring with feedback for learn-
ers, even when minimal speech data of L2 learners is available.
While the mobile app demonstration is designed for Finnish, the
app can also be tested in English.
Original language | English |
---|---|
Title of host publication | Interspeech 2024 |
Publisher | International Society for Computers and Their Applications (ISCA) |
Pages | 5212-5213 |
Number of pages | 2 |
Publication status | Published - 1 Sept 2024 |
MoE publication type | A4 Conference publication |
Event | Interspeech - Kos Island, Greece Duration: 1 Sept 2024 → 5 Sept 2024 |
Publication series
Name | Interspeech |
---|---|
ISSN (Electronic) | 2308-457X |
Conference
Conference | Interspeech |
---|---|
Country/Territory | Greece |
City | Kos Island |
Period | 01/09/2024 → 05/09/2024 |
Keywords
- Automatic Speech Assessment
- L2 speaking
- content feedback
- low-resource language
- mobile app
- LLM
Fingerprint
Dive into the research topics of 'CaptainA self-study mobile app for practising speaking: task completion assessment and feedback with generative AI'. Together they form a unique fingerprint.-
AASIS: Automatic assessment of spoken interaction in second language
Kurimo, M. (Principal investigator)
01/09/2023 → 31/08/2027
Project: RCF Academy Project
-
TEFLON: Technology-enhanced foreign and second-language learning of Nordic languages
Kurimo, M. (Principal investigator)
01/04/2021 → 31/03/2025
Project: Unknown
-
DigiTala: Aka-Digi Tala
Kurimo, M. (Principal investigator)
01/01/2020 → 31/08/2023
Project: RCF Academy Project targeted call