Projects per year
Abstract
When evaluating and comparing models using leaveoneout crossvalidation (LOOCV), the uncertainty of the estimate is typically assessed using the variance of the sampling distribution. Considering the uncertainty is important, as the variability of the estimate can be high in some cases. Previous studies show that no general unbiased variance estimator can be constructed, that would apply for any utility or loss measure and any model. We show that it is possible to construct an unbiased estimator considering a specific predictive performance measure and model. We demonstrate an unbiased sampling distribution variance estimator for the Bayesian normal model with fixed model variance using the expected log pointwise predictive density (elpd) utility score. This example demonstrates that it is possible to obtain improved, problemspecific, unbiased estimators for assessing the uncertainty in LOOCV estimation.
Original language  English 

Number of pages  24 
Journal  COMMUNICATIONS IN STATISTICS: THEORY AND METHODS 
Early online date  12 Jan 2022 
DOIs  
Publication status  Published  3 Feb 2022 
MoE publication type  A1 Journal articlerefereed 
Keywords
 Bayesian computation
 leaveoneout crossvalidation
 uncertainty
 variance estimator
 bias
Fingerprint
Dive into the research topics of 'Unbiased estimator for the variance of the leaveoneout crossvalidation estimator for a Bayesian normal model with fixed variance'. Together they form a unique fingerprint.Projects
 3 Finished

FCAI: Finnish Center for Artificial Intelligence
01/01/2019 → 31/12/2022
Project: Academy of Finland: Other research funding

Reliable Automated Bayesian Machine Learning
01/01/2018 → 31/12/2019
Project: Academy of Finland: Other research funding

Computational methods for survival analysis
Vehtari, A., Andersen, M. & Siivola, E.
01/09/2016 → 31/08/2020
Project: Academy of Finland: Other research funding