Unbiased estimator for the variance of the leave-one-out cross-validation estimator for a Bayesian normal model with fixed variance

Research output: Contribution to journalArticleScientificpeer-review

Abstract

When evaluating and comparing models using leave-one-out cross-validation (LOO-CV),
the uncertainty of the estimate is typically assessed using the variance of the sampling distribution.
It is known, however, that no unbiased estimator for the variance can be constructed in a general case.
While it has not been discussed before, it could be possible to construct such an estimator by considering specific models.
In this paper, we show that an unbiased sampling distribution variance estimator is obtainable for the Bayesian normal model with fixed model variance using expected log pointwise predictive density (elpd) utility score.
Instead of the obtained pointwise LOO-CV estimates, we estimate the variance directly from the observations.
Motivated by the presented unbiased variance estimator, it could be possible to obtain other improved problem-specific estimators, not only unbiased ones, for assessing the uncertainty of LOO-CV estimation.
Original languageEnglish
Number of pages22
JournalarXiv.org
Publication statusSubmitted - 21 Sep 2020
MoE publication typeA1 Journal article-refereed

Keywords

  • Bayesian computation
  • leave-one-out cross-validation
  • uncertainty
  • variance estimator
  • bias

Fingerprint Dive into the research topics of 'Unbiased estimator for the variance of the leave-one-out cross-validation estimator for a Bayesian normal model with fixed variance'. Together they form a unique fingerprint.

Cite this