Evaluating Retrieval-Augmented Generation-Large Language Models for Infective Endocarditis Prophylaxis: Clinical Accuracy and Efficiency
Abstract
INTRODUCTION AND AIMS: The use of large language models (LLMs) in healthcare is expanding. Retrieval-augmented generation (RAG) addresses key LLM limitations by grounding responses in domain-specific, up-to-date information. This study evaluated RAG-augmented LLMs for infective endocarditis (IE) prophylaxis in dental procedures, comparing their performance with non-RAG models assessed in our previous publication using the same question set. A pilot study also explored the utility of an LLM as a clinical decision support tool.
METHODS: An established IE prophylaxis question set from previous research was used to ensure comparability. Ten LLMs integrated with RAG were tested using MiniLM L6 v2 embeddings and FAISS to retrieve relevant content from the 2021 American Heart Association IE guideline. Models were evaluated across five independent runs, with and without a preprompt ('You are an experienced dentist'), a prompt-engineering technique used in previous research to improve LLMs accuracy. Three RAG-LLMs were compared to their native (non-RAG) counterparts benchmarked in the previous study. In the pilot study, 10 dental students (5 undergraduate, 5 postgraduate in oral and maxillofacial surgery) completed the questionnaire unaided, then again with assistance from the best performing LLM. Accuracy and task time were measured.
RESULTS: DeepSeek Reasoner achieved the highest mean accuracy (83.6%) without preprompting, while Grok 3 beta reached 90.0% with preprompting. The…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.