Learning Constructions of Natural Language: Statistical Models and Evaluations

Sami Virpioja

Learning Constructions of Natural Language: Statistical Models and Evaluations

Julkaisun otsikon käännös: Luonnollisen kielen rakenteiden oppiminen: tilastollisia malleja ja evaluaatiomenetelmiä

Sami Virpioja

Tutkimustuotos: Doctoral Thesis › Collection of Articles

Abstrakti

Luonnollisen kielen automaattinen käsittely pohjautuu yhä suuremmassa määrin tilastollisten koneoppimismenetelmien käyttöön jatkuvasti lisääntyvälle elektroniselle teksti- ja puheaineistolle. Tyypillisiä sovelluksia tilatollisille menetelmille ovat esimerkiksi tiedonhaku, puheentunnistus ja konekäännös. Monet sovellusten osaongelmat ovat ratkaistavissa ilman kieliriippuvaisia resursseja, kuten annotoituja aineistoja, käyttämällä ohjaamatonta koneoppimista. Tämä väitöskirja keskittyy erityisesti yhteen tällaiseen ongelmaan: leksikaalisten perusyksiköiden valintaan. Käytettävien yksiköiden valinta on tekstiaineiston käsittelyn ensimmäinen askel ja edeltää esimerkiksi kielimallien estimointia tai vektoriesitysten laskemista. Perinteisiä ratkaisuja yksiköiden valintaan ovat yksinkertaiset heuristiikat sekä kieliopilliset sääntöpohjaiset työkalut. Niiden sijaan tässä työssä esitetään datalähtöistä, ohjaamattomaan oppimiseen perustuvaa lähestymistapaa yksiköiden valintaan. Sen etuina ovat joustavuus ja riippumattomuus siitä, mitä lingvistisiä resursseja halutulle kielelle ja sovellusalueelle on saatavilla. Koska tilastollisesti opitut yksiköt eivät aina osu yhteen perinteisten kielitieteellisten perusluokkien kanssa, niitä kutsutaan tässä työssä konstruktioiksi. Termi pohjautuu konstruktiokielioppeihin, jotka ovat käyttöpohjaisia, kognitiivisia teorioita kielestä. Väitöskirjassa esitetyt menetelmät konstruktioiden oppimiseen perustuvat Morfessor-nimiseen menetelmään, joka mallintaa morfologista pilkontaa tilastollisesti ja ohjaamattomasti. Uudet menetelmät käsittelevät allomorfian oppimista, morfologisen pilkonnan osittain ohjattua oppimista sekä lausetason konstruktioiden oppimista. Saadut tulokset ovat kilpailukykyisiä erityisesti morfologian oppimisessa. Työssä esitellään myös uusia tekniikoita Morfessorin tuottamien morfologisten konstruktioiden käyttöön tilastollisessa kielenmallinnuksessa ja konekäännöksessä. Käytännön sovellusten ohella Morfessorin osoitetaan toimivan myös psykolingvistisen datan mallinnuksessa: sen todennäköisyysestimaatit sanoille korreloivat vahvasti ihmisten reaktioaikoihin leksikaalisessa päätöksenteossa. Lisäksi väitöskirjassa tutkitaan kielen ohjaamattoman oppimisen suoria evaluaatiomenetelmiä. Suora evaluaatio, esimerkiksi algoritmin tulosten vertaaminen olemassa oleviin kieliopillisiin annotaatioihin, on usein nopeampaa ja yksinkertaisempaa kuin epäsuora evaluaatio kielenkäsittelyn sovellusten toiminnan kautta. Ohjaamattoman oppimisen tapauksessa vertailu annotoituun dataan ei kuitenkaan aina ole suoraviivaista. Tässä väitöskirjassa kehitetään evaluaatiomenetelmiä erityisesti kahteen ongelmaan: sanojen morfologian oppimiseen ja vektorimuotoisten dokumenttiesitysten oppimiseen. Molemmissa on haasteena löytää moniulotteisesta datasta yhteydet eri piirreparien välille. Ehdotetut menetelmät ovat nopeita käyttää ja ne ennustavat hyvin sovelluksista saatuja tuloksia.

Julkaisun otsikon käännös	Luonnollisen kielen rakenteiden oppiminen: tilastollisia malleja ja evaluaatiomenetelmiä
Alkuperäiskieli	Englanti
Pätevyys	Tohtorintutkinto
Myöntävä instituutio	Aalto-yliopisto
Valvoja/neuvonantaja	Oja, Erkki, Vastuuprofessori Kurimo, Mikko, Ohjaaja Lagus, Krista, Ohjaaja
Kustantaja	Aalto University
Painoksen ISBN	978-952-60-4882-6
Sähköinen ISBN	978-952-60-4883-3
Tila	Julkaistu - 2012
OKM-julkaisutyyppi	G5 Artikkeliväitöskirja

Tutkimusalat

morfeemipilkonta
morfologian oppiminen
konstruktiokielioppi
ohjaamaton oppiminen
osittain ohjattu oppiminen
todennäköisyysmallit
kielimallit
vektoriavaruusmallit
konekäännös
puheentunnistus

Pääsy asiakirjaan

http://urn.fi/URN:ISBN:978-952-60-4883-3

Siteeraa tätä

@phdthesis{80e787e2863842bd85079785f2a87bbd,

title = "Learning Constructions of Natural Language: Statistical Models and Evaluations",

abstract = "The modern, statistical approach to natural language processing relies on using machine learning techniques on the increasing amount of text and speech data in electronic format. Typical applications for statistical methods include information retrieval, speech recognition, and machine translation. Many problems encountered in the applications can be solved without language-dependent resources, such as annotated data sets, by the means of unsupervised learning. This thesis focuses on one such problem: the selection of lexical units. It is the first step in processing text data, preceding, for example, the estimation of language models or extraction of vectorial representations. While the lexical units are often selected using simple heuristics or grammatical rule-based methods, this thesis proposes the use of unsupervised and semi-supervised machine learning. Advantages of the data-driven unit selection include greater flexibility and independence from the linguistic resources that exist for a particular language and domain. Statistically learned lexical units do not always fit to the categories in traditional linguistic theories. In this thesis, they are called constructions according to construction grammars, a family of usage-based, cognitive theories of grammar. For learning constructions of a language, the thesis builds on Morfessor, an unsupervised statistical method for morphological segmentation. Morfessor is successfully extended to the tasks of learning allomorphs, semi-supervised learning of morphological segmentation, and learning phrasal constructions of sentences. The results are competitive especially for the morphology induction problems. The thesis also includes new techniques for using the sub-word constructions learned by Morfessor in statistical language modeling and machine translation. In addition to its usefulness in the applications, Morfessor is shown to have psycholinguistic competence: its probability estimates have high correlations with human reaction times in a lexical decision task. Furthermore, direct evaluation methods for the unit selection and other learning problems are considered. Direct evaluations, such as comparing the output of the algorithm to existing linguistic annotations, are often quicker and simpler than indirect evaluation via the end-user applications. However, with unsupervised algorithms, the comparison to the reference data is not always straightforward. In this thesis, direct evaluation methods are developed for two unsupervised tasks, morphology induction and learning semantic vector representations of documents. In both cases, the challenge is to find relationships between the pairs of features in multidimensional data. The proposed methods are quick to use and they can accurately predict the performance in different applications.",

keywords = "morpheme segmentation, morphology induction, construction grammar, unsupervised learning, semi-supervised learning, probabilistic models, language models, vector space models, machine translation, speech recognition, morfeemipilkonta, morfologian oppiminen, konstruktiokielioppi, ohjaamaton oppiminen, osittain ohjattu oppiminen, todenn{\"a}k{\"o}isyysmallit, kielimallit, vektoriavaruusmallit, konek{\"a}{\"a}nn{\"o}s, puheentunnistus, morpheme segmentation, morphology induction, construction grammar, unsupervised learning, semi-supervised learning, probabilistic models, language models, vector space models, machine translation, speech recognition",

author = "Sami Virpioja",

year = "2012",

language = "English",

isbn = "978-952-60-4882-6",

series = "Aalto University publication series DOCTORAL DISSERTATIONS",

publisher = "Aalto University",

number = "158",

address = "Finland",

school = "Aalto University",

}

TY - BOOK

T1 - Learning Constructions of Natural Language: Statistical Models and Evaluations

AU - Virpioja, Sami

PY - 2012

Y1 - 2012

N2 - The modern, statistical approach to natural language processing relies on using machine learning techniques on the increasing amount of text and speech data in electronic format. Typical applications for statistical methods include information retrieval, speech recognition, and machine translation. Many problems encountered in the applications can be solved without language-dependent resources, such as annotated data sets, by the means of unsupervised learning. This thesis focuses on one such problem: the selection of lexical units. It is the first step in processing text data, preceding, for example, the estimation of language models or extraction of vectorial representations. While the lexical units are often selected using simple heuristics or grammatical rule-based methods, this thesis proposes the use of unsupervised and semi-supervised machine learning. Advantages of the data-driven unit selection include greater flexibility and independence from the linguistic resources that exist for a particular language and domain. Statistically learned lexical units do not always fit to the categories in traditional linguistic theories. In this thesis, they are called constructions according to construction grammars, a family of usage-based, cognitive theories of grammar. For learning constructions of a language, the thesis builds on Morfessor, an unsupervised statistical method for morphological segmentation. Morfessor is successfully extended to the tasks of learning allomorphs, semi-supervised learning of morphological segmentation, and learning phrasal constructions of sentences. The results are competitive especially for the morphology induction problems. The thesis also includes new techniques for using the sub-word constructions learned by Morfessor in statistical language modeling and machine translation. In addition to its usefulness in the applications, Morfessor is shown to have psycholinguistic competence: its probability estimates have high correlations with human reaction times in a lexical decision task. Furthermore, direct evaluation methods for the unit selection and other learning problems are considered. Direct evaluations, such as comparing the output of the algorithm to existing linguistic annotations, are often quicker and simpler than indirect evaluation via the end-user applications. However, with unsupervised algorithms, the comparison to the reference data is not always straightforward. In this thesis, direct evaluation methods are developed for two unsupervised tasks, morphology induction and learning semantic vector representations of documents. In both cases, the challenge is to find relationships between the pairs of features in multidimensional data. The proposed methods are quick to use and they can accurately predict the performance in different applications.

AB - The modern, statistical approach to natural language processing relies on using machine learning techniques on the increasing amount of text and speech data in electronic format. Typical applications for statistical methods include information retrieval, speech recognition, and machine translation. Many problems encountered in the applications can be solved without language-dependent resources, such as annotated data sets, by the means of unsupervised learning. This thesis focuses on one such problem: the selection of lexical units. It is the first step in processing text data, preceding, for example, the estimation of language models or extraction of vectorial representations. While the lexical units are often selected using simple heuristics or grammatical rule-based methods, this thesis proposes the use of unsupervised and semi-supervised machine learning. Advantages of the data-driven unit selection include greater flexibility and independence from the linguistic resources that exist for a particular language and domain. Statistically learned lexical units do not always fit to the categories in traditional linguistic theories. In this thesis, they are called constructions according to construction grammars, a family of usage-based, cognitive theories of grammar. For learning constructions of a language, the thesis builds on Morfessor, an unsupervised statistical method for morphological segmentation. Morfessor is successfully extended to the tasks of learning allomorphs, semi-supervised learning of morphological segmentation, and learning phrasal constructions of sentences. The results are competitive especially for the morphology induction problems. The thesis also includes new techniques for using the sub-word constructions learned by Morfessor in statistical language modeling and machine translation. In addition to its usefulness in the applications, Morfessor is shown to have psycholinguistic competence: its probability estimates have high correlations with human reaction times in a lexical decision task. Furthermore, direct evaluation methods for the unit selection and other learning problems are considered. Direct evaluations, such as comparing the output of the algorithm to existing linguistic annotations, are often quicker and simpler than indirect evaluation via the end-user applications. However, with unsupervised algorithms, the comparison to the reference data is not always straightforward. In this thesis, direct evaluation methods are developed for two unsupervised tasks, morphology induction and learning semantic vector representations of documents. In both cases, the challenge is to find relationships between the pairs of features in multidimensional data. The proposed methods are quick to use and they can accurately predict the performance in different applications.

KW - morpheme segmentation

KW - morphology induction

KW - construction grammar

KW - unsupervised learning

KW - semi-supervised learning

KW - probabilistic models

KW - language models

KW - vector space models

KW - machine translation

KW - speech recognition

KW - morfeemipilkonta

KW - morfologian oppiminen

KW - konstruktiokielioppi

KW - ohjaamaton oppiminen

KW - osittain ohjattu oppiminen

KW - todennäköisyysmallit

KW - kielimallit

KW - vektoriavaruusmallit

KW - konekäännös

KW - puheentunnistus

KW - morpheme segmentation

KW - morphology induction

KW - construction grammar

KW - unsupervised learning

KW - semi-supervised learning

KW - probabilistic models

KW - language models

KW - vector space models

KW - machine translation

KW - speech recognition

M3 - Doctoral Thesis

SN - 978-952-60-4882-6

T3 - Aalto University publication series DOCTORAL DISSERTATIONS

PB - Aalto University

ER -

Learning Constructions of Natural Language: Statistical Models and Evaluations

Abstrakti

Tutkimusalat

Pääsy asiakirjaan

Sormenjälki

Siteeraa tätä