Repeated knockout tournaments for efficient screening of the top-ranking sample

In this paper, we investigate how a repeated knockout tournament design could be applied to psychoacoustic experimentation. By simulating tournaments consisting of pairwise comparisons between players (cf. sound samples), we investigate the statistics of the outcomes of knockout (KO) and round-robin (RR) tournaments under various realistic assumptions regarding the pairwise winning probabilities. We find that often the KO design can identify the top-ranking player in fewer total games but with similar accuracy as the RR. We suggest that knockout tournament designs can be beneficial especially when the aim is to narrow down on a small subset of best samples among a large pool of initial alternatives.