Evaluation Metrics for Retrieval-Augmented Generation (RAG) Systems: A Framework Proposal
Abstract
Abstract: Retrieval-Augmented Generation (RAG) has established itself as a primary method for connecting large language models (LLMs) to external knowledge sources while addressing hallucination problems and enhancing factual accuracy [1,4,6]. Current evaluation approaches remain fragmented, as they often assess retrieval relevance [9,12] and surface-level text quality [2,5] independently, without linking these perspectives across the complete pipeline. This work introduces a multi-level evaluation framework that systematically decomposes assessment into retrieval, generation, and integration stages. The retrieval layer is assessed using ranking and coverage metrics, including Precision@k, Recall@k, Mean Reciprocal Rank (MRR), and NDCG [3,10]. The generation layer incorporates lexical and semantic measures such as BLEU [14], ROUGE-L [15], BERTScore [16], and claim-based faithfulness [7,11]. At the integration layer, this paper proposes the Hallucination Attribution Index (HAI), which quantifies the proportion of generated claims grounded in retrieved evidence. Additionally, the integration of consistency testing and human-in-the-loop evaluations [8,13] strengthens system robustness in adversarial and domain-sensitive scenarios. By combining these elements, the proposed framework provides a comprehensive evaluation system for RAG that enables precise assessment and improves deployment trustworthiness across domains such as healthcare, law, and finance.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.