Multiplicity adjustment approaches in publicly funded multi-arm trials: a comprehensive review of the National Institute for Health and Care Research (NIHR) Journals Library

Background

Parallel-group multi-arm trials are randomised controlled trials (RCTs) where participants are allocated to three or more concurrent treatment groups. Multiplicity occurs when several statistical tests are conducted within the same study. Statistical adjustments to the design and analysis of multi-arm trials can be used to control the study-wise type I error rate. There is no clear guidance or consensus on the necessity of multiplicity adjustment in multi-arm trials, nor on which methods are most appropriate. This comprehensive review aimed to investigate the design, analysis and reporting of publicly funded parallel-group multi-arm trials and to report the approach to multiplicity in these trials with respect to sample size and statistical analysis.

Methods

We searched the United Kingdom’s National Institute for Health and Care Research (NIHR) online Journals Library, from 1 January 1997 to 31 December 2024 for reports of multi-arm RCTs. Information on the trial characteristics, the sample size estimation and analysis of the primary outcome was extracted. Two researchers conducted the search and selected reports for inclusion. Data from each report was independently extracted by two reviewers, and any disagreement was resolved by discussion.

Results

A total of 2452 reports, published online in the NIHR Journals Library, were screened for eligibility; 97 reports of multi-arm parallel-group trials met the inclusion criteria. Of these, 90 included the results of a multi-arm efficacy analysis. In the review, 35% (34/97) of the trials did adjust for multiplicity in the sample size calculation; in 84% (76/90), the potential between-arm comparisons were described in the methods, and 37% (33/90) made a multiplicity adjustment in the analysis. A further 86% (77/86) reported 95% confidence intervals. For the minority of multi-arm trials that did adjust for multiplicity, the most common adjustment method was Bonferroni.

Conclusions

The majority of the publicly funded multi-arm trials did not adjust for multiplicity in the sample size, statistical analysis, or estimation of confidence intervals. Researchers should follow the Consolidated Standards of Reporting Trials guidelines for multi-arm trials and clearly state in protocols and trial reports whether a multiplicity adjustment was made or provide a reason if no adjustment was made.