Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews
Abstract
Objectives This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews. Methods A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the “Cochrane RCT Classifier”), with the algorithm trained using a data set of title–abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification. Results The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98–0.99) and precision of 0.08 (95% confidence interval 0.06–0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published. Conclusions The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a…
How strong is the evidence?
Evidence Hub reads this as a randomized trial, done in people. Its evidence level is 2 of 5 (High · randomized trial). Randomized trials of a reasonable size are the strongest single-study design.
Read the paper
Related studies
- Challenges and opportunities with drug repurposing: finding strategies to find alternative uses of therapeutics
- Challenges in Chagas Disease Drug Development
- T Cell Specificity: A Great Challenge in Chagas Disease
- Benznidazole treatment leads to DNA damage in Trypanosoma cruzi and the persistence of rare widely dispersed non-replicative amastigotes in mice
- Rapid Molecular Detection of Terbinafine-resistant Dermatophytes
- Identifying novel drugs with new modes of action for neglected tropical fungal skin diseases (fungal skinNTDs) using an Open Source Drug discovery approach
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.