RADAR: A Prototype for Grounding Large Language Model Consult Support in an ACR-Structured Appropriateness Knowledge Base with Citation Validation and Abstention
Abstract
Large language models (LLMs) answer medical questions fluently but fabricate statements and citations too often to be trusted unaided at the point of care. We describe RADAR (Radiology Appropriateness Decision and Advisory Resource), a prototype retrieval-augmented consult-support pipeline that grounds an LLM in a knowledge base structured like the American College of Radiology (ACR) Appropriateness Criteria at the level of individual clinical variants. The knowledge base is synthetic (twelve variants across five topics) because the official criteria are copyrighted. The design is deliberately conservative: a BM25 retriever, an absolute-score abstention gate, a structured-output generator bound by an in-prompt grounding contract, and a deterministic validator that strips any recommendation whose citation does not resolve to a retrieved chunk. We report a development set (eight scenarios, seven off-domain probes) used during parameter selection and a held-out set written after all parameters were frozen (36 scenarios, three per variant, and 24 clinically adjacent probes), evaluated in a deterministic retrieval-only mode and in the full pipeline over five independent runs at temperature 0. Retrieval alone ranked the correct variant first in 25 of 36 held-out scenarios (69.4%; 95% CI 51.9-83.7) but always placed it within the top five, and the score gate stopped only 3 of 22 clinically adjacent probes. The full pipeline (claude-sonnet-4-6, temperature 0, five runs) selected the…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.