Evaluating Accuracy in Large Language Models: Benchmarking Corrective Rag Vs. Naive Retrieval Augmented Generation Approach
Abstract
Retrieval-Augmented Generation (RAG) has emerged as a promising approach to mitigate the limitations of Large Language Models (LLMs) in generating factually accurate and consistent text. The main focus of this technical survey is on correct answers as the key performance indicator (KPI) for comparing two well-known RAG methods: Naive RAG and Corrective retrieval augmented Generation (CRAG). Naive RAG exhibits a strong dependence on the relevance of retrieved documents, resulting in suboptimal performance when retrieval quality is compromised. CRAG, on the other hand, adds new features to improve robustness and adaptability, such as a retrieval evaluator, large-scale web searches, and a decompose-then-recompose algorithm. We introduce the Comprehensive RAG Benchmark (CRAG), which encompasses a diverse set of question-answer pairs spanning multiple domains, categories, entity popularities, and temporal dynamics, to facilitate a comprehensive evaluation of RAG models' performance in generating correct answers. Experiments show that adding RAG to LLMs makes a big difference in how many correct answers you get, with CRAG consistently beating Naive RAG. Nevertheless, RAG models, including CRAG, demonstrate lower answer correctness when confronted with questions pertaining to highly dynamic, less popular, or more complex facts. These results make it clear that more research and development is needed to make RAG models more reliable and able to give correct answers in a wider range…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.