What is the Test-Retest Reliability of Common Task-fMRI Measures? New Empirical Evidence and a Meta-Analysis

Identifying brain biomarkers of disease risk and treatment response is a growing priority in neuroscience. The ability to identify meaningful biomarkers is fundamentally limited by measurement reliability; measures that do not yield reliable values are unsuitable as biomarkers to predict clinical outcomes. Measuring brain activity using task-fMRI is a major focus of biomarker development; however, the reliability of task-fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. First, a meta-analysis of 90 experiments with 1,088 participants reporting 1,146 ICCs for task-fMRI revealed poor overall reliability (mean ICC=.397). Second, the test-retest reliabilities of activity in a priori regions of interest across 11 commonly used fMRI tasks collected in the Human Connectome Project and the Dunedin Longitudinal Study were poor (ICCs=.067-.485). Collectively, these findings demonstrate that commonly used task-fMRI measures are not currently suitable for brain biomarker discovery or individual differences research in cognitive neuroscience (i.e., brain-behavior mapping). We review how this state of affairs came to be and consider several avenues for improving the reliability of task-fMRI.