Evaluation Metrics for Retrieval Augmented Generation in the Scientific Domain
Abstract
Retrieval-Augmented Generation represents the state-of-the-art approach to perform questionanswering (QA) tasks in the scientific domain. This system combines a powerful generative component, capable of producing grammatically sound and readable answers, with a retrieval component that efficiently locates specific information within a large corpus of documents. As such, RAG systems are particularly well-suited to address the complexities inherent in this task. However, evaluating the accuracy and quality of the generated answers remains a significant challenge. The aim of this thesis was to find an effective method for assessing RAG performance in a scientific QA task. To this end, we conducted an extensive review of the current automatic evaluation metrics in use. The most common approach involves comparing generated answers with a reference produced by humans. Such comparison can focus on the form (lexical similarity), the content (semantic similarity), or on a deeper analysis through the use of Large Language Models (model-based). Each of these approaches has well-documented advantages and drawbacks, making it necessary to rigorously test their reliability and effectiveness in this context. To explore this, we selected representative metrics from each category and designed three progressively complex experiments to challenge them and analyze their behavior: 1) Can the metrics distinguish between correct and incorrect answers? 2) Can they differentiate between answers of…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.