User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Standard

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction. / Daee, Pedram; Peltola, Tomi; Vehtari, Aki; Kaski, Samuel.

IUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces. Association for Computing Machinery (ACM), 2018. p. 305-310.

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Harvard

Daee, P, Peltola, T, Vehtari, A & Kaski, S 2018, User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction. in IUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces. Association for Computing Machinery (ACM), pp. 305-310, International Conference on Intelligent User Interfaces , Tokyo, Japan, 07/03/2018. https://doi.org/10.1145/3172944.3172989

APA

Daee, P., Peltola, T., Vehtari, A., & Kaski, S. (2018). User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction. In IUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces (pp. 305-310). Association for Computing Machinery (ACM). https://doi.org/10.1145/3172944.3172989

Vancouver

Daee P, Peltola T, Vehtari A, Kaski S. User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction. In IUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces. Association for Computing Machinery (ACM). 2018. p. 305-310 https://doi.org/10.1145/3172944.3172989

Author

Daee, Pedram ; Peltola, Tomi ; Vehtari, Aki ; Kaski, Samuel. / User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction. IUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces. Association for Computing Machinery (ACM), 2018. pp. 305-310

Bibtex - Download

@inproceedings{f9928c2ecd00470692594def8736a1e6,
title = "User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction",
abstract = "In human-in-the-loop machine learning, the user provides information beyond that in the training data. Many algorithms and user interfaces have been designed to optimize and facilitate this human--machine interaction; however, fewer studies have addressed the potential defects the designs can cause. Effective interaction often requires exposing the user to the training data or its statistics. The design of the system is then critical, as this can lead to double use of data and overfitting, if the user reinforces noisy patterns in the data. We propose a user modelling methodology, by assuming simple rational behaviour, to correct the problem. We show, in a user study with 48 participants, that the method improves predictive performance in a sparse linear regression sentiment analysis task, where graded user knowledge on feature relevance is elicited. We believe that the key idea of inferring user knowledge with probabilistic user models has general applicability in guarding against overfitting and improving interactive machine learning.",
keywords = "Interactive machine learning, Probabilistic modeling, Bayesian Inference, overfitting, expert prior elicitation, human-in-the-loop machine learning",
author = "Pedram Daee and Tomi Peltola and Aki Vehtari and Samuel Kaski",
year = "2018",
month = "3",
day = "8",
doi = "10.1145/3172944.3172989",
language = "English",
pages = "305--310",
booktitle = "IUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces",
publisher = "Association for Computing Machinery (ACM)",
address = "United States",

}

RIS - Download

TY - GEN

T1 - User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

AU - Daee, Pedram

AU - Peltola, Tomi

AU - Vehtari, Aki

AU - Kaski, Samuel

PY - 2018/3/8

Y1 - 2018/3/8

N2 - In human-in-the-loop machine learning, the user provides information beyond that in the training data. Many algorithms and user interfaces have been designed to optimize and facilitate this human--machine interaction; however, fewer studies have addressed the potential defects the designs can cause. Effective interaction often requires exposing the user to the training data or its statistics. The design of the system is then critical, as this can lead to double use of data and overfitting, if the user reinforces noisy patterns in the data. We propose a user modelling methodology, by assuming simple rational behaviour, to correct the problem. We show, in a user study with 48 participants, that the method improves predictive performance in a sparse linear regression sentiment analysis task, where graded user knowledge on feature relevance is elicited. We believe that the key idea of inferring user knowledge with probabilistic user models has general applicability in guarding against overfitting and improving interactive machine learning.

AB - In human-in-the-loop machine learning, the user provides information beyond that in the training data. Many algorithms and user interfaces have been designed to optimize and facilitate this human--machine interaction; however, fewer studies have addressed the potential defects the designs can cause. Effective interaction often requires exposing the user to the training data or its statistics. The design of the system is then critical, as this can lead to double use of data and overfitting, if the user reinforces noisy patterns in the data. We propose a user modelling methodology, by assuming simple rational behaviour, to correct the problem. We show, in a user study with 48 participants, that the method improves predictive performance in a sparse linear regression sentiment analysis task, where graded user knowledge on feature relevance is elicited. We believe that the key idea of inferring user knowledge with probabilistic user models has general applicability in guarding against overfitting and improving interactive machine learning.

KW - Interactive machine learning

KW - Probabilistic modeling

KW - Bayesian Inference

KW - overfitting

KW - expert prior elicitation

KW - human-in-the-loop machine learning

UR - https://arxiv.org/abs/1710.04881

UR - https://github.com/HIIT/human-overfitting-in-IML

U2 - 10.1145/3172944.3172989

DO - 10.1145/3172944.3172989

M3 - Conference contribution

SP - 305

EP - 310

BT - IUI 2018 - Proceedings of the 23rd International Conference on Intelligent User Interfaces

PB - Association for Computing Machinery (ACM)

ER -

ID: 15805147