Developing an AI-assisted Low-resource Spoken Language Learning App for Children

Yaroslav Getman, Nhan Phan, Ragheb Al-Ghezi, Ekaterina Voskoboinik, Mittul Singh, Tamas Grosz, Mikko Kurimo, Giampiero Salvi, Torbjorn Svendsen, Sofia Strombergsson, Anna Smolander, Sari Ylinen

Research output: Contribution to journalArticleScientificpeer-review

13 Downloads (Pure)

Abstract

Computer-assisted Language Learning (CALL) is a rapidly developing area accelerated by advancements in the field of AI. A well-designed and reliable CALL system allows students to practice language skills, like pronunciation, any time outside of the classroom. Furthermore, gamification via mobile applications has shown encouraging results on learning outcomes and motivates young users to practice more and perceive language learning as a positive experience. In this work, we adapt the latest speech recognition technology to be a part of an online pronunciation training system for small children. As part of our gamified mobile application, our models will assess the pronunciation quality of young Swedish children diagnosed with Speech Sound Disorder, and participating in speech therapy. Additionally, the models provide feedback to young non-native children learning to pronounce Swedish and Finnish words. Our experiments revealed that these new models fit into an online game as they function as speech recognizers and pronunciation evaluators simultaneously. To make our systems more trustworthy and explainable, we investigated whether the combination of modern input attribution algorithms and time-aligned transcripts can explain the decisions made by the models, give us insights into how the models work and provide a tool to develop more reliable solutions.

Original languageEnglish
Pages (from-to)86025-86037
Number of pages13
JournalIEEE Access
Volume11
DOIs
Publication statusPublished - 2023
MoE publication typeA1 Journal article-refereed

Keywords

  • Artificial intelligence
  • ASR
  • children’s speech
  • Computer aided diagnosis
  • Hidden Markov models
  • L2 speech
  • Pediatrics
  • Recording
  • speech rating
  • Speech recognition
  • SSD
  • Task analysis
  • Toy manufacturing industry
  • Training
  • Vocabulary
  • wav2vec2

Fingerprint

Dive into the research topics of 'Developing an AI-assisted Low-resource Spoken Language Learning App for Children'. Together they form a unique fingerprint.

Cite this