Developing a high-stake digital spoken language proficiency assessment: Results from pilot tests

The DigiTala project aims at developing a prototype of a technology-assisted high-stake spoken language proficiency assessment using multidisciplinary expertise from language education, signal processing and phonetics. The prototype will use automatic speech recognition (ASR) tool specially developed for educational second language (L2) use. The first target application in Finland will be the new digital matriculation examination, where language tests are recorded and digitized for assessments. The objectives are to enhance the reliability and practicality of oral language testing in a high-stakes context by clarifying the phonetic realizations of assessment criteria and developing methods that enable the utilization of ASR in the assessment process. A database of all recordings and assessments will provide a unique research opportunity for scrutinizing oral proficiency.

The study integrates modern technologies and principles of agile development with a traditional examination system to develop a stakeholder-friendly assessment. Students and their teachers from five secondary schools in Finland have participated in the study. A group of ten experts on the target language will assess the fluency and pronunciation of speech samples from more than 200 students. The assessment criteria is based on the Common European Framework of Reference. Automatic assessments will be applied when the ASR tool is trained for L2 purposes.

This article will present the assessment application developed through the one-year project and the results from questionnaires collected from secondary school students and teachers who participated in pilot tests. The article will also report findings on the reliabilities on teacher and automatic assessments. The results will provide important information about the benefits as well as obstacles and challenges of a digitized assessment system.