Building personalised speech technology systems with sparse, bad quality or out-of-domain data

Reima Karhila

Building personalised speech technology systems with sparse, bad quality or out-of-domain data

Julkaisun otsikon käännös: Building personalised speech technology systems with sparse, bad quality or out-of-domain data

Reima Karhila

Speech Recognition

Tutkimustuotos: Doctoral Thesis › Collection of Articles

Abstrakti

Automaattisen puheentunnistuksen ja puhesynteesin avulla on mahdollista rakentaa käyttöliittymiä, jotka eivät vaadi käsien tai silmien käyttöä. Näillä käyttöliittymillä voi ohjata sovelluksia tietokoneissa, puhelimissa, kodinelektroniikassa tai puettavassa elektroniikassa. Kuulijan havainnot synteettisen äänen laadusta ja puhujaidentiteetistä ovat olennainen osa käyttäjäkokemusta. Erilaisten puhujaidentiteettien kirjo on käytännössä rajaton, mikäli puhujaidentiteetin siirtoon voidaan käyttää lyhyitä tai muihin tarkoituksiin kerättyjä puheaineistoja. Tässä väitöskirjassa kuvataan tausta, menetelmät sekä tulokset joukolle tilastollisen parametrisen puhesynteesin ja puheentunnistuken alojen kokeita. Kokeiden pääpaino on puhujamallien adaptoinnissa ja järjestelmien tuotosten laadun arvioinissa. Kaikki kuvatut järjestelmät nojaavat puhemalleihin, jotka on opetettu suuresta puhe- ja tekstiaineistosta. Puheaineistot on esikäsitelty akustisiksi piirrevektoreiksi vokooderilla. Saatavilla olevan datan määrän ja laadun vaikutuksia tutkitaan väitöskirjassa raportoiduissa kokeissa, jotka koskevat äänitysten taustamelun vaikutuksia puhuja-adaptoitavaan paramtetiseen HMM-GMM puhesynteesiin, kuulijoiden kokemusta puhujaidentiteetistä rajatulla datamäärällä kielen yli adaptoivassa puhesynteesissä ja yksittäisen puhujan ja puhujajoukon adaptaatioiden ketjuttamista puhesynteesin parantamiseksi uudentyyppisille puhujille, jotka eivät ole hyvin edustettuin opetusdatassa. Puhuja-adaptaation tekemistä kielen yli tutkitaan kokeissa, jotka koskevat probabilistista puhuja-adaptaatiota kun saatavilla ei ole riittävää määrää lähtökielen puhedataa ja kaksikielistä puhesynteesiä lauseensisäisellä koodinvaihdolla kun käytössä ei ole toisen kielen puhedataa puhujalta. Kaikissa näissä kokeissa puhujaidentiteetti onnistuttiin siirtämään synteettiselle äänelle, vaikka adaptaatiodata oli meluisaa, toisella kielellä tai sitä oli hyvin vähän. Kielen yli adaptointia tutkittiin myös kaksikielisten puhujien puheentunnistuksessa, ja sen huomattiin parantavan tunnistustuloksia. Minkä tahansa järjestelmän kehittäminen vaatii tuotosten laadun mittaamista, ja tässä väitöskirjassa on mukana taustaa synteettisen puheen sekä kielenoppijoiden luonnollisen puheen objektiivisisesta sekä subjektiivisesta arvioinnista, ja myös analyysi erilaisten objektiivisten mittojen käytöstä HMM-GMM-pohjaisen puhesynteesin laadun arviointiin. Väitöskirjassa esitellään myös puhesynteesi- ja puheentunnistuskomponentteihin perustuva järjestelmä, joka arvioi ja pisteyttää vieraan kielen oppijoiden lausumisyrityksiä. Yksittäisten lausumien automaattinen arvostelu on hankalaa. Järjestelmämme onnistui tekemään riittävän tarkkoja arvioita riittävän nopeasti, jotta sitä voitiin käyttää kielenoppimisen tutkimukseen kehitetyissä tietokonepeleissä.

Julkaisun otsikon käännös	Building personalised speech technology systems with sparse, bad quality or out-of-domain data
Alkuperäiskieli	Englanti
Pätevyys	Tohtorintutkinto
Myöntävä instituutio	Aalto-yliopisto
Valvoja/neuvonantaja	Kurimo, Mikko, Vastuuprofessori Kurimo, Mikko, Ohjaaja
Kustantaja	Aalto University
Painoksen ISBN	978-952-60-8594-4
Sähköinen ISBN	978-952-60-8595-1
Tila	Julkaistu - 2019
OKM-julkaisutyyppi	G5 Artikkeliväitöskirja

Tutkimusalat

tilastollinen parametrinen puhesynteesi
automaattinen puheentunnistus
synteettisen puheen laadunarviointi
tietokoneavusteinen lausumisenopetus
akustisten mallien adaptointi

Pääsy asiakirjaan

http://urn.fi/URN:ISBN:978-952-60-8595-1

Akustiikan laboratorio
Ville Pulkki (Manager)
Sähkötekniikan korkeakoulu
Laitteistot/tilat: Facility
Science-IT
Mikko Hakala (Manager)
Perustieteiden korkeakoulu
Laitteistot/tilat: Facility

Siteeraa tätä

@phdthesis{657d6452daea48b2bada45227e9bc7b1,

title = "Building personalised speech technology systems with sparse, bad quality or out-of-domain data",

abstract = "Automatic speech recognition and text-to-speech systems offer handsfree and eyesfree interfaces for applications on computers, telephones and home and wearable electronics. The perceived quality and identity of a text-to-speech system's voice are essential to the user experience. The possibilities for different speaker identities are practically limitless if short or out-of-domain collections of speech can be used to transfer speaker identity to the synthetic voice. This thesis describes background, methods and results for a group of experiments performed with statistical parametric speech synthesis and speech recognition, with focus on speaker adaptation of the models and evaluation the quality of the systems' output. All these systems rely on speech models that are trained on large collections of speech and text data. The speech data have been preprocessed into acoustic features using a vocoder. The amount and quality of available data are addressed in experiments on the effects of background noise in the adaptation data of speaker-adaptive HMM-GMM statistical parametric speech synthesis, listener perception of speaker background in speaker adapted speech synthesis with sparse, foreign-accented data, and stacking group and speaker adaptations to improve quality of speech synthesis for out-of-domain speakers. Cross-lingual adaptation is investigated in experiments on probabilistic cross-lingual speaker adaptation when a model for source language is not available, and bilingual speech synthesis with code-switching when source language data is not available for one of the languages. In all these studies, the speaker characteristics were successfully transferred to a synthesic voice even if the adaptation data was noisy, in another language or there was very little of it. Cross-lingual adaptation was also investigated for automatic speech recognition of bilingual speakers and found to improve recognition results. Any system development relies on measuring the quality of the output, and this thesis also includes an overview of objective and subjective methods of quality evaluation for synthetic speech and natural foreign language speech, as well as an analysis of different objective measures for evaluating quality of HMM-GMM based speech synthesis systems. Building on components of speech recognition and synthesis systems, this thesis also presents a system for evaluating and scoring the pronunciation quality of foreign language learners utterances. Rating pronunciation quality of single utterances is a difficult problem but our system manages to do it at a speed and reliability that is satisfactory for computer games used to study language learning.",

keywords = "statistical parametric speech synthesis, automatic speech recognition, computer assisted pronunciation training, speech synthesis quality evaluation, acoustic model adaptation, tilastollinen parametrinen puhesynteesi, automaattinen puheentunnistus, synteettisen puheen laadunarviointi, tietokoneavusteinen lausumisenopetus, akustisten mallien adaptointi, statistical parametric speech synthesis, automatic speech recognition, computer assisted pronunciation training, speech synthesis quality evaluation, acoustic model adaptation",

author = "Reima Karhila",

year = "2019",

language = "English",

isbn = "978-952-60-8594-4",

series = "Aalto University publication series DOCTORAL DISSERTATIONS",

publisher = "Aalto University",

number = "110",

address = "Finland",

school = "Aalto University",

}

TY - BOOK

T1 - Building personalised speech technology systems with sparse, bad quality or out-of-domain data

AU - Karhila, Reima

PY - 2019

Y1 - 2019

N2 - Automatic speech recognition and text-to-speech systems offer handsfree and eyesfree interfaces for applications on computers, telephones and home and wearable electronics. The perceived quality and identity of a text-to-speech system's voice are essential to the user experience. The possibilities for different speaker identities are practically limitless if short or out-of-domain collections of speech can be used to transfer speaker identity to the synthetic voice. This thesis describes background, methods and results for a group of experiments performed with statistical parametric speech synthesis and speech recognition, with focus on speaker adaptation of the models and evaluation the quality of the systems' output. All these systems rely on speech models that are trained on large collections of speech and text data. The speech data have been preprocessed into acoustic features using a vocoder. The amount and quality of available data are addressed in experiments on the effects of background noise in the adaptation data of speaker-adaptive HMM-GMM statistical parametric speech synthesis, listener perception of speaker background in speaker adapted speech synthesis with sparse, foreign-accented data, and stacking group and speaker adaptations to improve quality of speech synthesis for out-of-domain speakers. Cross-lingual adaptation is investigated in experiments on probabilistic cross-lingual speaker adaptation when a model for source language is not available, and bilingual speech synthesis with code-switching when source language data is not available for one of the languages. In all these studies, the speaker characteristics were successfully transferred to a synthesic voice even if the adaptation data was noisy, in another language or there was very little of it. Cross-lingual adaptation was also investigated for automatic speech recognition of bilingual speakers and found to improve recognition results. Any system development relies on measuring the quality of the output, and this thesis also includes an overview of objective and subjective methods of quality evaluation for synthetic speech and natural foreign language speech, as well as an analysis of different objective measures for evaluating quality of HMM-GMM based speech synthesis systems. Building on components of speech recognition and synthesis systems, this thesis also presents a system for evaluating and scoring the pronunciation quality of foreign language learners utterances. Rating pronunciation quality of single utterances is a difficult problem but our system manages to do it at a speed and reliability that is satisfactory for computer games used to study language learning.

AB - Automatic speech recognition and text-to-speech systems offer handsfree and eyesfree interfaces for applications on computers, telephones and home and wearable electronics. The perceived quality and identity of a text-to-speech system's voice are essential to the user experience. The possibilities for different speaker identities are practically limitless if short or out-of-domain collections of speech can be used to transfer speaker identity to the synthetic voice. This thesis describes background, methods and results for a group of experiments performed with statistical parametric speech synthesis and speech recognition, with focus on speaker adaptation of the models and evaluation the quality of the systems' output. All these systems rely on speech models that are trained on large collections of speech and text data. The speech data have been preprocessed into acoustic features using a vocoder. The amount and quality of available data are addressed in experiments on the effects of background noise in the adaptation data of speaker-adaptive HMM-GMM statistical parametric speech synthesis, listener perception of speaker background in speaker adapted speech synthesis with sparse, foreign-accented data, and stacking group and speaker adaptations to improve quality of speech synthesis for out-of-domain speakers. Cross-lingual adaptation is investigated in experiments on probabilistic cross-lingual speaker adaptation when a model for source language is not available, and bilingual speech synthesis with code-switching when source language data is not available for one of the languages. In all these studies, the speaker characteristics were successfully transferred to a synthesic voice even if the adaptation data was noisy, in another language or there was very little of it. Cross-lingual adaptation was also investigated for automatic speech recognition of bilingual speakers and found to improve recognition results. Any system development relies on measuring the quality of the output, and this thesis also includes an overview of objective and subjective methods of quality evaluation for synthetic speech and natural foreign language speech, as well as an analysis of different objective measures for evaluating quality of HMM-GMM based speech synthesis systems. Building on components of speech recognition and synthesis systems, this thesis also presents a system for evaluating and scoring the pronunciation quality of foreign language learners utterances. Rating pronunciation quality of single utterances is a difficult problem but our system manages to do it at a speed and reliability that is satisfactory for computer games used to study language learning.

KW - statistical parametric speech synthesis

KW - automatic speech recognition

KW - computer assisted pronunciation training

KW - speech synthesis quality evaluation

KW - acoustic model adaptation

KW - tilastollinen parametrinen puhesynteesi

KW - automaattinen puheentunnistus

KW - synteettisen puheen laadunarviointi

KW - tietokoneavusteinen lausumisenopetus

KW - akustisten mallien adaptointi

KW - statistical parametric speech synthesis

KW - automatic speech recognition

KW - computer assisted pronunciation training

KW - speech synthesis quality evaluation

KW - acoustic model adaptation

M3 - Doctoral Thesis

SN - 978-952-60-8594-4

T3 - Aalto University publication series DOCTORAL DISSERTATIONS

PB - Aalto University

ER -

Building personalised speech technology systems with sparse, bad quality or out-of-domain data

Abstrakti

Tutkimusalat

Pääsy asiakirjaan

Sormenjälki

Laitteet

Akustiikan laboratorio

Science-IT

Siteeraa tätä