Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis
Abstract
Background Large language models (LLMs) have flourished and gradually become an important research and application direction in the medical field. However, due to the high degree of specialization, complexity, and specificity of medicine, which results in extremely high accuracy requirements, controversy remains about whether LLMs can be used in the medical field. More studies have evaluated the performance of various types of LLMs in medicine, but the conclusions are inconsistent. Objective This study uses a network meta-analysis (NMA) to assess the accuracy of LLMs when answering clinical research questions to provide high-level evidence-based evidence for its future development and application in the medical field. Methods In this systematic review and NMA, we searched PubMed, Embase, Web of Science, and Scopus from inception until October 14, 2024. Studies on the accuracy of LLMs when answering clinical research questions were included and screened by reading published reports. The systematic review and NMA were conducted to compare the accuracy of different LLMs when answering clinical research questions, including objective questions, open-ended questions, top 1 diagnosis, top 3 diagnosis, top 5 diagnosis, and triage and classification. The NMA was performed using Bayesian frequency theory methods. Indirect intercomparisons between programs were performed using a grading scale. A larger surface under the cumulative ranking curve (SUCRA) value indicates a higher ranking…
How strong is the evidence?
Evidence Hub reads this as a meta-analysis. Its evidence level is 1 of 5 (Highest · pooled evidence). Pooled evidence from several studies sits at the top of the evidence pyramid.
Read the paper
Related studies
- Improving sleep quality leads to better mental health: A meta-analysis of randomised controlled trials
- Tocilizumab in patients with severe COVID-19: a retrospective cohort study
- A new aging measure captures morbidity and mortality risk across diverse subpopulations from NHANES IV: A cohort study
- Fruit, vegetable, and legume intake, and cardiovascular disease and deaths in 18 countries (PURE): a prospective cohort study
- Are noise and air pollution related to the incidence of dementia? A cohort study in London, England
- Long COVID symptoms and duration in SARS-CoV-2 positive children — a nationwide cohort study
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.