Automating clinical trial outcome identification and misreporting detection using RegCheck
Abstract
Objective: To evaluate the accuracy and cost of RegCheck, an automated large language model (LLM)-based workflow, for identifying clinical trial outcomes and detecting outcome misreporting by comparing its outputs with manual assessments from the COMPare Trials project.
Design: Validation study.
Setting: Sixty-two clinical trials originally assessed in the COMPare Trials project, sampled from five high impact general medical journals.
Participants: Published clinical trial reports and their corresponding prespecified registrations and/or protocols.
Main outcome measures: Four prespecified research questions were examined. RQ1 assessed outcome extraction recall relative to COMPare. RQ2 assessed accuracy of outcome classification as primary, secondary, or non-prespecified. RQ3 assessed accuracy of misreporting detection relative to COMPare, with additional manual adjudication of discrepancies between RegCheck and COMPare. RQ4 assessed the average per-paper cost of running the automated workflow.
Results: Across the validation papers, RegCheck achieved 91.2% outcome extraction recall relative to COMPare, and 83.6% outcome classification accuracy. For detection of outcome misreporting, RegCheck’s overall accuracy was 85.6%. However, after resolving discrepancies with the original human judgements (which frequently favoured RegCheck’s judgement), revised accuracy for outcome misreporting detection was 94.8%. The mean cost of running the workflow was 5.94 USD per paper.…
Read the paper
Related studies
- Comparative effectiveness of GLP-1 receptor agonists on glycaemic control, body weight, and lipid profile for type 2 diabetes: systematic review and network meta-analysis
- Adverse effects of cyclooxygenase 2 inhibitors on renal and arrhythmia events: meta-analysis of randomized trials
- Benefits and harms of drug treatment for type 2 diabetes: systematic review and network meta-analysis of randomised controlled trials
- Comparative effectiveness and safety of statins as a class and of specific statins for primary prevention of cardiovascular disease: A systematic review, meta‐analysis, and network meta‐analysis of randomized trials with 94,283 participants
- Screen time increases overweight and obesity risk among adolescents: a systematic review and dose-response meta-analysis
- Exposure to WHO AWaRe antibiotics and isolation of multi-drug resistant bacteria: a systematic review and meta-analysis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.