Active Learning for Decision-Making from Imbalanced Observational Data

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Standard

Active Learning for Decision-Making from Imbalanced Observational Data. / Sundin, Iiris; Schulam, Peter; Siivola, Eero; Vehtari, Aki; Saria, Suchi; Kaski, Samuel.

36th International Conference on Machine Learning, ICML 2019. 2019. p. 10578-10587 (Proceedings of Machine Learning Research; Vol. 97).

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Harvard

Sundin, I, Schulam, P, Siivola, E, Vehtari, A, Saria, S & Kaski, S 2019, Active Learning for Decision-Making from Imbalanced Observational Data. in 36th International Conference on Machine Learning, ICML 2019. Proceedings of Machine Learning Research, vol. 97, pp. 10578-10587, International Conference on Machine Learning, Long Beach, United States, 09/06/2019.

APA

Sundin, I., Schulam, P., Siivola, E., Vehtari, A., Saria, S., & Kaski, S. (2019). Active Learning for Decision-Making from Imbalanced Observational Data. In 36th International Conference on Machine Learning, ICML 2019 (pp. 10578-10587). (Proceedings of Machine Learning Research; Vol. 97).

Vancouver

Sundin I, Schulam P, Siivola E, Vehtari A, Saria S, Kaski S. Active Learning for Decision-Making from Imbalanced Observational Data. In 36th International Conference on Machine Learning, ICML 2019. 2019. p. 10578-10587. (Proceedings of Machine Learning Research).

Author

Sundin, Iiris ; Schulam, Peter ; Siivola, Eero ; Vehtari, Aki ; Saria, Suchi ; Kaski, Samuel. / Active Learning for Decision-Making from Imbalanced Observational Data. 36th International Conference on Machine Learning, ICML 2019. 2019. pp. 10578-10587 (Proceedings of Machine Learning Research).

Bibtex - Download

@inproceedings{a315d2794aac4e0aa5bf649665848da1,
title = "Active Learning for Decision-Making from Imbalanced Observational Data",
abstract = "Machine learning can help personalized decision support by learning models to predict individual treatment effects (ITE). This work studies the reliability of prediction-based decision-making in a task of deciding which action a to take for a target unit after observing its covariates x~ and predicted outcomes p^(y~∣x~,a). An example case is personalized medicine and the decision of which treatment to give to a patient. A common problem when learning these models from observational data is imbalance, that is, difference in treated/control covariate distributions, which is known to increase the upper bound of the expected ITE estimation error. We propose to assess the decision-making reliability by estimating the ITE model’s Type S error rate, which is the probability of the model inferring the sign of the treatment effect wrong. Furthermore, we use the estimated reliability as a criterion for active learning, in order to collect new (possibly expensive) observations, instead of making a forced choice based on unreliable predictions. We demonstrate the effectiveness of this decision-making aware active learning in two decision-making tasks: in simulated data with binary outcomes and in a medical dataset with synthetic and continuous treatment outcomes.",
author = "Iiris Sundin and Peter Schulam and Eero Siivola and Aki Vehtari and Suchi Saria and Samuel Kaski",
year = "2019",
language = "English",
series = "Proceedings of Machine Learning Research",
publisher = "PMLR",
pages = "10578--10587",
booktitle = "36th International Conference on Machine Learning, ICML 2019",

}

RIS - Download

TY - GEN

T1 - Active Learning for Decision-Making from Imbalanced Observational Data

AU - Sundin, Iiris

AU - Schulam, Peter

AU - Siivola, Eero

AU - Vehtari, Aki

AU - Saria, Suchi

AU - Kaski, Samuel

PY - 2019

Y1 - 2019

N2 - Machine learning can help personalized decision support by learning models to predict individual treatment effects (ITE). This work studies the reliability of prediction-based decision-making in a task of deciding which action a to take for a target unit after observing its covariates x~ and predicted outcomes p^(y~∣x~,a). An example case is personalized medicine and the decision of which treatment to give to a patient. A common problem when learning these models from observational data is imbalance, that is, difference in treated/control covariate distributions, which is known to increase the upper bound of the expected ITE estimation error. We propose to assess the decision-making reliability by estimating the ITE model’s Type S error rate, which is the probability of the model inferring the sign of the treatment effect wrong. Furthermore, we use the estimated reliability as a criterion for active learning, in order to collect new (possibly expensive) observations, instead of making a forced choice based on unreliable predictions. We demonstrate the effectiveness of this decision-making aware active learning in two decision-making tasks: in simulated data with binary outcomes and in a medical dataset with synthetic and continuous treatment outcomes.

AB - Machine learning can help personalized decision support by learning models to predict individual treatment effects (ITE). This work studies the reliability of prediction-based decision-making in a task of deciding which action a to take for a target unit after observing its covariates x~ and predicted outcomes p^(y~∣x~,a). An example case is personalized medicine and the decision of which treatment to give to a patient. A common problem when learning these models from observational data is imbalance, that is, difference in treated/control covariate distributions, which is known to increase the upper bound of the expected ITE estimation error. We propose to assess the decision-making reliability by estimating the ITE model’s Type S error rate, which is the probability of the model inferring the sign of the treatment effect wrong. Furthermore, we use the estimated reliability as a criterion for active learning, in order to collect new (possibly expensive) observations, instead of making a forced choice based on unreliable predictions. We demonstrate the effectiveness of this decision-making aware active learning in two decision-making tasks: in simulated data with binary outcomes and in a medical dataset with synthetic and continuous treatment outcomes.

UR - https://arxiv.org/abs/1904.05268

UR - https://github.com/IirisSundin/active-learning-for-decision-making

M3 - Conference contribution

T3 - Proceedings of Machine Learning Research

SP - 10578

EP - 10587

BT - 36th International Conference on Machine Learning, ICML 2019

ER -

ID: 34076603