Neural waveform generation for source-filter vocoding in speech synthesis

Lauri Juvela

Neural waveform generation for source-filter vocoding in speech synthesis

Julkaisun otsikon käännös: Puheen aaltomuotojen tuottaminen hermoverkoilla lähde-suodin mallissa

Lauri Juvela

Speech Communication Technology

Tutkimustuotos: Doctoral Thesis › Collection of Articles

Abstrakti

Puhesynteesi, keinotekoisen puheen luominen tekstistä, on eräs puhekommunikaatioteknologian perustavista ongelmista. Puhesynteesin tutkimus on kehittynyt varhaisista vaiheistaan ihmisen puheentuoton mallinnuksen kautta nykymuotoonsa, jossa synteesiä voidaan soveltaa lukuisiin käyttötarkoituksiin. Näitä sovelluksia ovat muun muassa, näytönlukijat, puhekyvyttömien apuvälineet ja kasvavassa määrin tekoälyn synteettinen puhe koneen ja ihmisen välisessä vuorovaikutuksessa. Viime vuosina tilastollinen mallipohjainen synteesi on kehittynyt huomattavasti syvien hermoverkkojen ansiosta ja synteesin luonnollisuus vastaakin jo lähes ihmispuhetta. Tärkeä tekijä tässä kehityksessä ovat hermoverkkoihin perustuvat aaltomuotoja syntetisoivat mallit, jotka ottavat perinteisen puhesynteesijärjestelmän vokooderin roolin. Laadullisesta kehityksestä huolimatta viimeaikaisten "neurovokooderien" ja perinteisten signaalinkäsittelyyn perustuvien mallien välillä on aukko sekä ymmärryksessämme että uusien algoritmien laskennallisessa tehokkuudessa. Tämän väitöskirjan keskeinen motivaatio on ollut yhdistää viimeisimpiä generatiivisia hermoverkkomalleja klassisiin signaalinkäsittelyn metodeihin, ja luoda algoritmisesti ymmärrettävää, korkealaatuista ja tehokasta puhesynteesiä. Erityisesti tämä väitöskirja keskittyy mallintamaan puheentuoton lähde-suodinmallin herätesignaalia käyttäen syviä hermoverkkoja. Koska olemassa olevat signaalinkäsittelytekniikat kykenevät varsin hyvin mallintamaan puheen spektriverhokäyrää ja ääntöväylän resonansseja parametrisesti, jää tehtäväksi olennaisesti tuottaa äänilähdettä vastaava jäännösheräte hermoverkkomallilla. Tämä väitöskirja kehittää uuden esitystavan jaksollisten aaltomuotojen mallintamiseksi ja soveltaa generatiivisia kilpailevia hermoverkkoja (eng. generative adversarial networks) näiden aaltomuotojen synteesiin. Lisäksi väitöskirja esittää WaveNet-pohjaisen herätemallin, joka yhdistetään eksplisiittisen suodinmalliin synteesissä. Lopuksi nämä kaksi mallityyppiä yhdistetään lähde-suodin hermoverkkosyntetisaattoriksi, joka kykenee hyödyntämään tehokasta rinnakkaislaskentaa ja voidaan kouluttaa integroituna osana syvää hermoverkkoa.

Julkaisun otsikon käännös	Puheen aaltomuotojen tuottaminen hermoverkoilla lähde-suodin mallissa
Alkuperäiskieli	Englanti
Pätevyys	Tohtorintutkinto
Myöntävä instituutio	Aalto-yliopisto
Valvoja/neuvonantaja	Alku, Paavo, Vastuuprofessori
Kustantaja	Aalto University
Painoksen ISBN	978-952-60-3909-1
Sähköinen ISBN	978-952-60-3910-7
Tila	Julkaistu - 2020
OKM-julkaisutyyppi	G5 Artikkeliväitöskirja

Tutkimusalat

puhesynteesi
syväoppivat hermoverkot
generatiiviset mallit

Pääsy asiakirjaan

http://urn.fi/URN:ISBN:978-952-60-3910-7

Akustiikan laboratorio
Pulkki, V. (Manager)
Sähkötekniikan korkeakoulu
Laitteistot/tilat: Facility
Science-IT
Hakala, M. (Manager)
Perustieteiden korkeakoulu
Laitteistot/tilat: Facility

Siteeraa tätä

@phdthesis{8bf88a85d01b4722b64317ab862b9c20,

title = "Neural waveform generation for source-filter vocoding in speech synthesis",

abstract = "Speech synthesis, or artificial generation of speech from any given text, has been one of the fundamental problems in speech communication technology. While early research on synthesis was driven by curiosity about the human voice production, modern speech synthesis has found many applications in screen readers, assistive devices and human-computer speech interfaces, to name a few. With the recent advances in statistical model based synthesis using neural networks, speech synthesis has reached an unprecedented level of naturalness and flexibility that will make possible many exciting future applications. A major contributor to the recent improvements has been the introduction of neural network waveform synthesis models, which take the role of a vocoder in a traditional speech synthesis system. However, a gap remains both in understanding and computational efficiency of the algorithms between the recent raw waveform neural vocoders and the classical model-based signal processing vocoders. A central motivation of the present dissertation has been to combine the emerging generative neural network models with classical speech signal processing concepts for efficient, high-quality synthesis that retains a degree of interpretability. Specifically, this dissertation focuses on neural network modeling of the excitation signal related to the source-filter model of human voice production. Since the present signal processing techniques for modeling the spectral envelope of the vocal tract are highly developed, the spectral envelope can be parameterized and used directly as a part of neural vocoding schemes. The remaining task is then to develop neural network models for the residual excitation signal. This dissertation presents an improved framework for representing residual excitation waveform in a pitch synchronous format, and applies generative adversarial networks for synthesizing these waveforms without a parametric aperiodicity model. Furthermore, it proposes an autoregressive WaveNet based excitation model, which only explicitly uses a spectral envelope model during synthesis. Finally, the two approaches are combined into a parallel-inference-capable source-filter synthesizer, which is trainable in an end-to-end fashion.",

keywords = "speech synthesis, deep learning, generative models, GAN, puhesynteesi, syv{\"a}oppivat hermoverkot, generatiiviset mallit, speech synthesis, deep learning, generative models, GAN",

author = "Lauri Juvela",

year = "2020",

language = "English",

isbn = "978-952-60-3909-1",

series = "Aalto University publication series DOCTORAL DISSERTATIONS",

publisher = "Aalto University",

number = "85",

address = "Finland",

school = "Aalto University",

}

TY - BOOK

T1 - Neural waveform generation for source-filter vocoding in speech synthesis

AU - Juvela, Lauri

PY - 2020

Y1 - 2020

N2 - Speech synthesis, or artificial generation of speech from any given text, has been one of the fundamental problems in speech communication technology. While early research on synthesis was driven by curiosity about the human voice production, modern speech synthesis has found many applications in screen readers, assistive devices and human-computer speech interfaces, to name a few. With the recent advances in statistical model based synthesis using neural networks, speech synthesis has reached an unprecedented level of naturalness and flexibility that will make possible many exciting future applications. A major contributor to the recent improvements has been the introduction of neural network waveform synthesis models, which take the role of a vocoder in a traditional speech synthesis system. However, a gap remains both in understanding and computational efficiency of the algorithms between the recent raw waveform neural vocoders and the classical model-based signal processing vocoders. A central motivation of the present dissertation has been to combine the emerging generative neural network models with classical speech signal processing concepts for efficient, high-quality synthesis that retains a degree of interpretability. Specifically, this dissertation focuses on neural network modeling of the excitation signal related to the source-filter model of human voice production. Since the present signal processing techniques for modeling the spectral envelope of the vocal tract are highly developed, the spectral envelope can be parameterized and used directly as a part of neural vocoding schemes. The remaining task is then to develop neural network models for the residual excitation signal. This dissertation presents an improved framework for representing residual excitation waveform in a pitch synchronous format, and applies generative adversarial networks for synthesizing these waveforms without a parametric aperiodicity model. Furthermore, it proposes an autoregressive WaveNet based excitation model, which only explicitly uses a spectral envelope model during synthesis. Finally, the two approaches are combined into a parallel-inference-capable source-filter synthesizer, which is trainable in an end-to-end fashion.

AB - Speech synthesis, or artificial generation of speech from any given text, has been one of the fundamental problems in speech communication technology. While early research on synthesis was driven by curiosity about the human voice production, modern speech synthesis has found many applications in screen readers, assistive devices and human-computer speech interfaces, to name a few. With the recent advances in statistical model based synthesis using neural networks, speech synthesis has reached an unprecedented level of naturalness and flexibility that will make possible many exciting future applications. A major contributor to the recent improvements has been the introduction of neural network waveform synthesis models, which take the role of a vocoder in a traditional speech synthesis system. However, a gap remains both in understanding and computational efficiency of the algorithms between the recent raw waveform neural vocoders and the classical model-based signal processing vocoders. A central motivation of the present dissertation has been to combine the emerging generative neural network models with classical speech signal processing concepts for efficient, high-quality synthesis that retains a degree of interpretability. Specifically, this dissertation focuses on neural network modeling of the excitation signal related to the source-filter model of human voice production. Since the present signal processing techniques for modeling the spectral envelope of the vocal tract are highly developed, the spectral envelope can be parameterized and used directly as a part of neural vocoding schemes. The remaining task is then to develop neural network models for the residual excitation signal. This dissertation presents an improved framework for representing residual excitation waveform in a pitch synchronous format, and applies generative adversarial networks for synthesizing these waveforms without a parametric aperiodicity model. Furthermore, it proposes an autoregressive WaveNet based excitation model, which only explicitly uses a spectral envelope model during synthesis. Finally, the two approaches are combined into a parallel-inference-capable source-filter synthesizer, which is trainable in an end-to-end fashion.

KW - speech synthesis

KW - deep learning

KW - generative models

KW - GAN

KW - puhesynteesi

KW - syväoppivat hermoverkot

KW - generatiiviset mallit

KW - speech synthesis

KW - deep learning

KW - generative models

KW - GAN

M3 - Doctoral Thesis

SN - 978-952-60-3909-1

T3 - Aalto University publication series DOCTORAL DISSERTATIONS

PB - Aalto University

ER -

Neural waveform generation for source-filter vocoding in speech synthesis

Abstrakti

Tutkimusalat

Pääsy asiakirjaan

Sormenjälki

Laitteet

Akustiikan laboratorio

Science-IT

Siteeraa tätä