Bayesian Predictive Inference and Feature Selection for High-Dimensional Data

Juho Piironen

Bayesian Predictive Inference and Feature Selection for High-Dimensional Data

Julkaisun otsikon käännös: Bayesilainen ennustava päättely ja piirrevalinta korkeaulotteisille aineistoille

Juho Piironen

Tutkimustuotos: Doctoral Thesis › Collection of Articles

Abstrakti

Tämä väitöskirja käsittelee bayesilaista tilastollista päättelyä ohjatuissa oppimistehtävissä, joissa havaintoja on niukasti, mutta piirteiden määrä on suuri. Työssä keskitytään kahteen osaongelmaan. Ensimmäinen näistä on jonkin mielenkiinnon kohteena olevan muuttujan ennustaminen. Toinen ongelma on piirrevalinta, jossa tarkoituksena on löytää vain pieni joukko piirteitä, jotka ovat merkityksellisiä ennusteiden kannalta. Monissa tapauksissa hyvä ennustetarkkuus voi olla arvokasta sinällään ja usein auttaa ymmärtämään havaintoaineistoa. Piirrevalinta voi edelleen parantaa mallin tulkittavuutta ja selitettävyyttä, mutta sillä voidaan saavuttaa myös säästöjä, mikäli suuren piirremäärän käyttöön liittyy kustannuksia. Valtaosa aiemmin ehdotetuista menetelmistä pyrkii ratkaisemaan molemmat ongelmat samanaikaisesti käyttäen estimointimenetelmää, jossa piirrevalinta saadaan varsinaisen ennustemallin sovittamisen sivutuotteena täysin tai lähes automaattisesti. Tässä työssä esitetään, että monissa tapauksissa voidaan päästä parempaan lopputulokseen, mikäli noudatetaan päätösteoreettisesti perusteltua kaksivaiheista lähestymistapaa. Tässä lähestymistavassa muodostetaan ensin malli, joka ennustaa hyvin, mutta joka mahdollisesti käyttää isoa määrää piirteitä. Piirrevalinta suoritetaan tämän jälkeen etsimällä pienin mahdollinen joukko piirteitä, joilla saavutetaan olennaisesti samanlaiset ennusteet kuin alkuperäisellä mallilla. Tätä niin kutsuttua projektiivista lähetysmistapaa on ehdotettu kirjallisuudessa jo kauan sitten, mutta menetelmä ei ole saanut ansaitsemaansa huomiota. Tämä menetelmä antaa paljon vapauksia ennustemallin rakentamiseen, koska mallintajan ei tässä vaiheessa tarvitse välittää piirrevalinnasta. Toisaalta piirrevalinta usein helpottuu huomattavasti, mikäli tässä vaiheessa voidaan hyödyntää aiemmin sovitettua tarkkaa ennustemallia ja käyttää tätä referenssinä. Työssä keskitytään pääasiassa yleistettyihin lineaarimalleihin. Ennusteongelman ratkaisemiseksi työssä esitetään uusia menetelmiä harvuutta ja regularisointia koskevan priori-informaation sisällyttämiseksi ennustemalliin. Näillä menetelmillä voidaan joissakin tapauksissa parantaa mallin ennustekykyä ja tehdä mallin posteriori-laskennasta robustimpaa. Nämä tekniikat tuovat myös lisää teoreettista ymmärrystä eräiden usein käytettyjen priorijakaumien ominaisuuksista. Työssä tutkitaan myös laskennallisesti tehokkaita dimension redusointitekniikoita nopeuttamaan ennustemallin sovitusta havaintoaineistoissa, joissa piirteitä on hyvin paljon. Lisäksi työssä ehdotetaan alkuperäiseen projektiiviseen piirrevalintamenetelmään useita metodologisia parannuksia, joilla laskenta saadaan nopeaksi ja tarkaksi aineistoille, joissa piirteiden määrä on hyvin suuri. Työssä tutkitaan alustavasti myös, kuinka projektiivinen muuttujavalinta voidaan toteuttaa epälineaarisille ja ei-parametrisille malleille kuten gaussisille prosesseille. Väitöskirjan kontribuutiot ovat täysin metodologisia, mutta esitettyjen tekniikoiden etuja havainnollistetaan esimerkkiaineistoilla useilta sovellusaloilta, erityisesti laskennallisesta genetiikasta.

Julkaisun otsikon käännös	Bayesilainen ennustava päättely ja piirrevalinta korkeaulotteisille aineistoille
Alkuperäiskieli	Englanti
Pätevyys	Tohtorintutkinto
Myöntävä instituutio	Aalto-yliopisto
Valvoja/neuvonantaja	Vehtari, Aki, Vastuuprofessori Vehtari, Aki, Ohjaaja
Kustantaja	Aalto University
Painoksen ISBN	978-952-60-8538-8
Sähköinen ISBN	978-952-60-8539-5
Tila	Julkaistu - 2019
OKM-julkaisutyyppi	G5 Artikkeliväitöskirja

Tutkimusalat

Bayesilaiset yleistetyt lineaarimallit
piirrevalinta
dimension redusointi

Pääsy asiakirjaan

http://urn.fi/URN:ISBN:978-952-60-8539-5

Siteeraa tätä

@phdthesis{1d565fbc396d40889f5c75653d40f5ca,

title = "Bayesian Predictive Inference and Feature Selection for High-Dimensional Data",

abstract = "This thesis discusses Bayesian statistical inference in supervised learning problems where the data are scarce but the number of features large. The focus is on two important tasks. The first one is the prediction of some target variable of interest. The other task is feature selection, where the goal is to identify a small subset of features which are relevant for the prediction. A good predictive accuracy is often intrinsically valuable and a means to understanding the data. Feature selection can further help to make the model easier to interpret and reduce future costs if there is a price associated with predicting with many features. Most traditional approaches try to solve both problems at once by formulating an estimation procedure that performs automatic or semiautomatic feature selection as a by-product of the predictive model fitting. This thesis argues that in many cases one can benefit from a decision theoretically justified two-stage approach. In this approach, one first constructs a model that predicts well but possibly uses many features. In the second stage, one then finds a minimal subset of features that can characterize the predictions of this model. The basic idea of this so called projective framework has been around for a long time but it has largely been overlooked in the statistics and machine learning community. This approach offers plenty of freedom for building an accurate prediction model as one does not need to care about feature selection at this point, and it turns out solving the feature selection problem often becomes substantially easier given an accurate prediction model that can be used as a reference. The thesis focuses mostly on generalized linear models. To solve the problem of predictive model construction, the thesis introduces novel methods for encoding prior information about sparsity and regularization into the model. These methods can in some cases help to improve the prediction accuracy and robustify the posterior inference, but they also advance the current theoretical understanding of the fundamental characteristics of some commonly used prior distributions. The thesis explores also computationally efficient dimension reduction techniques that can be used as shortcuts for predictive model construction when the number of features is very large. Furthermore, the thesis develops the existing projective feature selection method further so as to make the computation fast and accurate for large number of features. Finally, the thesis takes the initial steps towards extending this framework to nonlinear and nonparametric Gaussian process models. The contributions of this thesis are solely methodological, but the benefits of the proposed methods are illustrated using example datasets from various fields, in particular from computational genetics.",

keywords = "Bayesian generalized linear models, feature selection, dimension reduction, Bayesilaiset yleistetyt lineaarimallit, piirrevalinta, dimension redusointi, Bayesian generalized linear models, feature selection, dimension reduction",

author = "Juho Piironen",

year = "2019",

language = "English",

isbn = "978-952-60-8538-8",

series = "Aalto University publication series DOCTORAL DISSERTATIONS",

publisher = "Aalto University",

number = "84",

address = "Finland",

school = "Aalto University",

}

TY - BOOK

T1 - Bayesian Predictive Inference and Feature Selection for High-Dimensional Data

AU - Piironen, Juho

PY - 2019

Y1 - 2019

N2 - This thesis discusses Bayesian statistical inference in supervised learning problems where the data are scarce but the number of features large. The focus is on two important tasks. The first one is the prediction of some target variable of interest. The other task is feature selection, where the goal is to identify a small subset of features which are relevant for the prediction. A good predictive accuracy is often intrinsically valuable and a means to understanding the data. Feature selection can further help to make the model easier to interpret and reduce future costs if there is a price associated with predicting with many features. Most traditional approaches try to solve both problems at once by formulating an estimation procedure that performs automatic or semiautomatic feature selection as a by-product of the predictive model fitting. This thesis argues that in many cases one can benefit from a decision theoretically justified two-stage approach. In this approach, one first constructs a model that predicts well but possibly uses many features. In the second stage, one then finds a minimal subset of features that can characterize the predictions of this model. The basic idea of this so called projective framework has been around for a long time but it has largely been overlooked in the statistics and machine learning community. This approach offers plenty of freedom for building an accurate prediction model as one does not need to care about feature selection at this point, and it turns out solving the feature selection problem often becomes substantially easier given an accurate prediction model that can be used as a reference. The thesis focuses mostly on generalized linear models. To solve the problem of predictive model construction, the thesis introduces novel methods for encoding prior information about sparsity and regularization into the model. These methods can in some cases help to improve the prediction accuracy and robustify the posterior inference, but they also advance the current theoretical understanding of the fundamental characteristics of some commonly used prior distributions. The thesis explores also computationally efficient dimension reduction techniques that can be used as shortcuts for predictive model construction when the number of features is very large. Furthermore, the thesis develops the existing projective feature selection method further so as to make the computation fast and accurate for large number of features. Finally, the thesis takes the initial steps towards extending this framework to nonlinear and nonparametric Gaussian process models. The contributions of this thesis are solely methodological, but the benefits of the proposed methods are illustrated using example datasets from various fields, in particular from computational genetics.

AB - This thesis discusses Bayesian statistical inference in supervised learning problems where the data are scarce but the number of features large. The focus is on two important tasks. The first one is the prediction of some target variable of interest. The other task is feature selection, where the goal is to identify a small subset of features which are relevant for the prediction. A good predictive accuracy is often intrinsically valuable and a means to understanding the data. Feature selection can further help to make the model easier to interpret and reduce future costs if there is a price associated with predicting with many features. Most traditional approaches try to solve both problems at once by formulating an estimation procedure that performs automatic or semiautomatic feature selection as a by-product of the predictive model fitting. This thesis argues that in many cases one can benefit from a decision theoretically justified two-stage approach. In this approach, one first constructs a model that predicts well but possibly uses many features. In the second stage, one then finds a minimal subset of features that can characterize the predictions of this model. The basic idea of this so called projective framework has been around for a long time but it has largely been overlooked in the statistics and machine learning community. This approach offers plenty of freedom for building an accurate prediction model as one does not need to care about feature selection at this point, and it turns out solving the feature selection problem often becomes substantially easier given an accurate prediction model that can be used as a reference. The thesis focuses mostly on generalized linear models. To solve the problem of predictive model construction, the thesis introduces novel methods for encoding prior information about sparsity and regularization into the model. These methods can in some cases help to improve the prediction accuracy and robustify the posterior inference, but they also advance the current theoretical understanding of the fundamental characteristics of some commonly used prior distributions. The thesis explores also computationally efficient dimension reduction techniques that can be used as shortcuts for predictive model construction when the number of features is very large. Furthermore, the thesis develops the existing projective feature selection method further so as to make the computation fast and accurate for large number of features. Finally, the thesis takes the initial steps towards extending this framework to nonlinear and nonparametric Gaussian process models. The contributions of this thesis are solely methodological, but the benefits of the proposed methods are illustrated using example datasets from various fields, in particular from computational genetics.

KW - Bayesian generalized linear models

KW - feature selection

KW - dimension reduction

KW - Bayesilaiset yleistetyt lineaarimallit

KW - piirrevalinta

KW - dimension redusointi

KW - Bayesian generalized linear models

KW - feature selection

KW - dimension reduction

M3 - Doctoral Thesis

SN - 978-952-60-8538-8

T3 - Aalto University publication series DOCTORAL DISSERTATIONS

PB - Aalto University

ER -

Bayesian Predictive Inference and Feature Selection for High-Dimensional Data

Abstrakti

Tutkimusalat

Pääsy asiakirjaan

Sormenjälki

Siteeraa tätä