Audited large language model triage for systematic review screening in national clinical guideline production: validation and prospective deployment
Abstract
Title and abstract screening limit the timeliness of systematic reviews used for clinical guidelines. We evaluated audited large language model (LLM) triage at Sweden's National Board of Health and Welfare. Ten LLMs from five model families were tested on 419 Cochrane reviews comprising 26,892 records, and the selected ensemble was externally validated on 133 reviews including 8,501 records matched to planned guideline topics. The same locked model pair was then used prospectively across 24 systematic reviews in two national guideline programmes. On the 419-review selection benchmark, the selected Gemini-3-flash plus GPT-5.1 ensemble achieved 98.0% (95% CI, 97.3-98.7) mean review-level sensitivity, while topic-matched validation yielded 96.7% sensitivity (95% CI, 93.7-98.9). Prospective deployment screened 74,679 records, placed 63,858 (85.5%) in the AI-excluded pool and reduced estimated first-pass screening effort from 415 to 34 person-days. Across 600 randomly sampled AI-excluded records from the migraine and dementia programmes, none was confirmed as a final false negative after post-unblinding adjudication; across the completed 680-record audit, all 38 final retained records had been AI flagged, whereas locked blinded human consensus missed seven. These findings support locked, audited LLM triage, with human oversight and programme-specific monitoring, for systematic reviews used in national guidelines.
How strong is the evidence?
Evidence Hub reads this as a systematic review. Its evidence level is 1 of 5 (Highest · pooled evidence). Pooled evidence from several studies sits at the top of the evidence pyramid.
Read the paper
Related studies
- Calcium Ions in the Physiology and Pathology of the Central Nervous System
- AI-powered in silico twins: redefining precision medicine through simulation, personalization, and predictive healthcare
- In silico functional, structural and pathogenicity analysis of missense single nucleotide polymorphisms in human MCM6 gene
- In Silico drug repurposing pipeline using deep learning and structure based approaches in epilepsy
- Targeting serotonin receptors with phytochemicals – an in-silico study
- In silico exploration of anti-prostate cancer compounds from differential expressed genes
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.