Influence of multiple hypothesis testing on reproducibility in neuroimaging research: A simulation study and Python-based software

Tuomas Puoliväli*, Satu Palva, J. Matias Palva

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

15 Citations (Scopus)


Background: Reproducibility of research findings has been recently questioned in many fields of science, including psychology and neurosciences. One factor influencing reproducibility is the simultaneous testing of multiple hypotheses, which entails false positive findings unless the analyzed p-values are carefully corrected. While this multiple testing problem is well known and studied, it continues to be both a theoretical and practical problem. New method: Here we assess reproducibility in simulated experiments in the context of multiple testing. We consider methods that control either the family-wise error rate (FWER) or false discovery rate (FDR), including techniques based on random field theory (RFT), cluster-mass based permutation testing, and adaptive FDR. Several classical methods are also considered. The performance of these methods is investigated under two different models. Results: We found that permutation testing is the most powerful method among the considered approaches to multiple testing, and that grouping hypotheses based on prior knowledge can improve power. We also found that emphasizing primary and follow-up studies equally produced most reproducible outcomes. Comparison with existing method(s): We have extended the use of two-group and separate-classes models for analyzing reproducibility and provide a new open-source software “MultiPy” for multiple hypothesis testing. Conclusions: Our simulations suggest that performing strict corrections for multiple testing is not sufficient to improve reproducibility of neuroimaging experiments. The methods are freely available as a Python toolkit “MultiPy” and we aim this study to help in improving statistical data analysis practices and to assist in conducting power and reproducibility analyses for new experiments.

Original languageEnglish
Article number108654
Number of pages11
JournalJournal of Neuroscience Methods
Publication statusPublished - 1 May 2020
MoE publication typeA1 Journal article-refereed


  • False discovery rate
  • Family-wise error rate
  • Multiple hypothesis testing
  • Neurophysiological data
  • Python
  • Reproducibility


Dive into the research topics of 'Influence of multiple hypothesis testing on reproducibility in neuroimaging research: A simulation study and Python-based software'. Together they form a unique fingerprint.

Cite this