CaptainA self-study mobile app for practising speaking: task completion assessment and feedback with generative AI

Nhan Phan Chi, Anna von Zansen, Maria Kautonen, Tamás Grósz, Mikko Kurimo

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

22 Downloads (Pure)

Abstract

We introduce the CaptainA mobile app, designed to meet
the needs of second language (L2) learners engaged in self-
study of Finnish, with potential applicability to other languages.
Our app can provide automatic speaking assessment (ASA) of
task completion in picture-based tasks, along with grading ex-
planations and corrective feedback. It can also automatically
generate pictures for visual tasks, providing users with unlim-
ited practice opportunities. The mobile app is based on our
framework that combines visual natural language generation
(NLG), automatic speech recognition (ASR), and prompting
large language model (LLM) for low-resource language. Our
goal is to promote the development of next-generation speech-
based computer-assisted language learning (CALL) systems ca-
pable of providing automatic scoring with feedback for learn-
ers, even when minimal speech data of L2 learners is available.
While the mobile app demonstration is designed for Finnish, the
app can also be tested in English.
Original languageEnglish
Title of host publicationInterspeech 2024
PublisherInternational Society for Computers and Their Applications (ISCA)
Pages5212-5213
Number of pages2
Publication statusPublished - 1 Sept 2024
MoE publication typeA4 Conference publication
EventInterspeech - Kos Island, Greece
Duration: 1 Sept 20245 Sept 2024

Publication series

NameInterspeech
ISSN (Electronic)2308-457X

Conference

ConferenceInterspeech
Country/TerritoryGreece
CityKos Island
Period01/09/202405/09/2024

Keywords

  • Automatic Speech Assessment
  • L2 speaking
  • content feedback
  • low-resource language
  • mobile app
  • LLM

Fingerprint

Dive into the research topics of 'CaptainA self-study mobile app for practising speaking: task completion assessment and feedback with generative AI'. Together they form a unique fingerprint.

Cite this