The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index
Abstract
OBJECTIVES
A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes.
STUDY DESIGN AND SETTING
A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result.
RESULTS
The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up.
CONCLUSION
The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results.
How strong is the evidence?
Evidence Hub reads this as a randomized trial, done in people. Its evidence level is 2 of 5 (High · randomized trial). Randomized trials of a reasonable size are the strongest single-study design. Reported size: 682 participants or records.
Read the paper
Related studies
- The impact of the COVID-19 epidemic on mental health of undergraduate students in New Jersey, cross-sectional study
- Viral suppression and HIV transmission in serodiscordant male couples: an international, prospective, observational, cohort study
- Lifetime risk of atrial fibrillation according to optimal, borderline, or elevated levels of risk factors: cohort study based on longitudinal data from the Framingham Heart Study
- Nusinersen in adults with 5q spinal muscular atrophy: a non-interventional, multicentre, observational cohort study
- Population-based rates, timing, and causes of maternal deaths, stillbirths, and neonatal deaths in south Asia and sub-Saharan Africa: a multi-country prospective cohort study
- 30-day mortality in patients hospitalized with COVID-19 during the first wave of the Italian epidemic: A prospective cohort study
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.