Measurement of retrieved chunk quality from real-world knowledge in retrieval-augmented generation: A Phase 1 foundational study
Abstract
Retrieval-augmented generation (RAG) holds promise for supporting high-stakes medical decision-making. However, most research has focused on downstream optimization of parameters and algorithms. This Phase 1 foundational study quantitatively evaluated the upstream quality of knowledge documents and their impact on retrieval performance, using Japanese clinical research protocol manuals for Institutional Review Board pre-screening support as a case study. We established a three-tier evaluation framework: Level 1 assessed knowledge document quality through independent expert review across Structure, Granularity, and Noise dimensions; Level 2a evaluated the structural quality of retrieved chunks using large language model-as-a-Judge across five metrics; and Level 2b conducted proof-of-concept content appropriateness evaluation against a Gold Standard derived from international guidelines. Using Google Cloud Vertex AI Search, we analyzed 594 chunks from baseline knowledge (A-line: four institutional manuals as-is) and six chunks from optimized knowledge (B-line: proof of concept). Level 2a evaluations employed deterministic settings with five independent trials, achieving excellent reliability (intraclass correlation coefficient of 0.936). The results revealed substantial quality limitations in the A-line chunks: the median scores were 2.0 or below across all five metrics, with fewer than 20% of the chunks reaching practical utility thresholds (score of 4 or higher). Even among…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.