Evidence Coverage Evaluator: A Comprehensive Framework for Assessing Grounding Quality in Retrieval-Augmented Generation Systems
Abstract
Retrieval-Augmented Generation (RAG) systems have emerged as a powerful paradigm for enhancing large language models with external knowledge. However, evaluating whether generated answers are properly grounded in retrieved evidence remains a critical challenge. Unlike accuracy-based metrics, evidence coverage evaluates whether each factual claim is explicitly supported by retrieved context, independent of surfacelevel correctness. This paper presents the Evidence Coverage Evaluator (ECE), a comprehensive framework for assessing evidence coverage in RAG-generated answers. ECE introduces two complementary evaluation modes: Mode A, a lightweight NLIbased approach using pre-trained natural language inference models (RoBERTa-MNLI) achieving 0.88-0.91 coverage scores in 2.1-2.8 seconds, and Mode B, an LLM-based judge mode using local Ollama models (Mistral, Llama3, Gemma3, DeepSeek-R1) achieving 0.86 coverage scores with 100% success rates in 23-31 seconds for top-performing models. Our framework extracts finegrained claims from answers, retrieves relevant evidence passages using BM25 or embedding-based methods, and scores claimevidence pairs to compute coverage metrics. Experimental results demonstrate that ECE outperforms baseline methods (keyword matching, BERTScore, direct NLI) by 10-32% in coverage assessment. Additionally, ECE includes advanced features such as citation quality assessment, unsupported claim identification, and actionable feedback generation. The framework is…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.