Can a Large Language Model Serve as the Missing Second Reviewer? Prominence Bias, Retrieval Effects, and a Confidence-Fabrication Gap in an Empirical Evaluation of Two Published Meta-Analyses
Abstract
Abstract Background Single-author systematic reviews (SRs) and meta-analyses (MAs) are common in the biomedical literature, yet methodological guidance discourages single authorship because a second independent reviewer reduces error and selection bias. Large language models (LLMs) have been proposed as a means of supplying that missing second reviewer, but existing validation studies are small (4–26 reviews) and largely single-condition. Objective To empirically evaluate two task designs — independent extraction from a known study list (“Design B”) and autonomous identification of studies from a research question alone (“Design A”) — and to characterize the specific ways an LLM second reviewer succeeds and fails. Methods A current-generation, publicly available LLM (Claude, Anthropic) was tested across repeated, independent sessions on two published meta-analyses selected for topical and temporal separation from one another: a metformin-and-cancer meta-analysis of 9 randomized controlled trials (RCTs) for Design B, and a multivessel-versus-culprit-only PCI meta-analysis of 9 RCTs for Design A. Design A was tested both without and with explicit retrieval-tool instructions. All model outputs were independently verified against primary sources by the author. Results In Design B (8 runs across two prompt-schema iterations), the model identified four real, independently verifiable errors in the published meta-analysis’s summary table, and separately produced one confirmed…
How strong is the evidence?
Evidence Hub reads this as a meta-analysis. Its evidence level is 1 of 5 (Highest · pooled evidence). Pooled evidence from several studies sits at the top of the evidence pyramid.
Read the paper
Related studies
- Metformin as an adjuvant in breast cancer treatment
- Metformin Lysosomal Targeting: A Novel Aspect to Be Investigated for Metformin Repurposing in Neurodegenerative Diseases?
- Aberrant HSF1 signaling activation underlies metformin amelioration of myocardial infarction in mice
- Metformin induces ZFP36 by mTORC1 inhibition in cervical cancer-derived cell lines
- Protein Interaction Network for Identifying Vascular Response of Metformin (Oral Antidiabetic)
- Preventing Long COVID With Metformin
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.