GuidelineGuard: An Agentic Retrieval-Augmented Generation Framework with Sentence-Level Citation Auditing for Guideline-Grounded Question Answering
Abstract
Background: Large language models (LLMs) can produce clinically plausible recommendations that are not adequately supported by authoritative evidence. Objectives: We introduce GuidelineGuard, a modular multi-agent retrieval-augmented generation pipeline in which a separate Auditor verifies claim–sentence support before a recommendation is surfaced. Methods: The original evaluation used a 73-sentence guideline corpus and GG-Bench-60, with replication across three open-weight backbones. In response to reviewer concerns about benchmark size and selective evaluation, we added a source-traceable GG-Bench-200 stress test and the complete 500-case held-out PQA-L test split of PubMedQA. The revision experiments compare single-pass RAG, a paired multi-agent no-Auditor ablation, and GuidelineGuard; the paired runner is designed to share the Planner–Retriever–Clinician draft so that the Auditor is the only intervention. Checkpoint verification confirmed an identical observable pre-audit state for all 200 GG-Bench cases and 496/500 PubMedQA cases; four PubMedQA cases were regenerated after quota-interrupted resumption and were correct commitments in both arms. Because the originally used hosted Llama endpoints became unavailable after the initial experiments, the expanded runs use openai/gpt-oss-20b for generation and openai/gpt-oss-120b for the Auditor. Results: On GG-Bench-200, single-pass RAG achieved 0.970 operational accuracy, while the no-Auditor and GuidelineGuard arms achieved…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.