Improving large language models accuracy for aortic stenosis treatment via Heart Team simulation: a prompt design analysis
Abstract
Abstract Aims Large language models (LLMs) have shown potential in clinical decision support, but the influence of prompt design on their performance, particularly in complex cardiology decision-making, is not well understood. Methods and results We retrospectively reviewed 231 patients evaluated by our Heart Team for severe aortic stenosis, with treatment options including surgical aortic valve replacement, transcatheter aortic valve implantation, or medical therapy. We tested multiple prompt-design strategies using zero-shot (0-shot), Chain-of-Thought (CoT), and Tree-of-Thought (ToT) prompting, combined with few-shot prompting, free/guided-thinking, and self-consistency. Patient data were condensed into standardized vignettes and queried using GPT4-o (version 2024–05–13, OpenAI) 40 times per patient under each prompt (147 840 total queries). Primary endpoint was mean accuracy; secondary endpoints included sensitivity, specificity, area under the curve (AUC), and treatment invasiveness. Guided-thinking-ToT achieved the highest accuracy (94.04%, 95% CI 90.87–97.21), significantly outperforming few-shot-ToT (87.16%, 95% CI 82.68–91.63) and few-shot-CoT (85.32%, 95% CI 80.59–90.06; P < 0.0001). Zero-shot prompting showed the lowest accuracy (73.39%, 95% CI 67.48–79.31). Guided-thinking-ToT yielded the highest AUC values (up to 0.97) and was the only prompt whose invasiveness did not differ significantly from Heart Team decisions (P = 0.078). An inverted quadratic relationship…
Read the paper
Related studies
- Global, regional, and national burden of hepatitis B, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019
- A meta‐analysis of anal cancer incidence by risk group: Toward a unified anal cancer risk scale
- Global fertility in 204 countries and territories, 1950–2021, with forecasts to 2100: a comprehensive demographic analysis for the Global Burden of Disease Study 2021
- Exercise interventions in Alzheimer's disease: A systematic review and meta-analysis of randomized controlled trials
- Prognostic Significance of Carbonic Anhydrase IX Expression in Cancer Patients: A Meta-Analysis
- Glucocorticoid Resistance: Is It a Requisite for Increased Cytokine Production in Depression? A Systematic Review and Meta-Analysis
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.