Evaluation of LLMs’ Reasoning for Retrieval Augmented Generation (RAG) Systems
Abstract
Large Language Models (LLMs) have shown remarkable capabilities in various Natural Language Processing Tasks, but are usually suffered by hallucinations which refers to generating factually incorrect content. Retrieval-Augmented Generation (RAG) has shown itself to be a promising approach for minimizing such hallucinations in by accessing external data sources. However, the impact of LLM’s reasoning capabilities such as Chain-of-thought (CoT) prompting in RAG systems is unclear. Three different LLMs, o1-mini, o3-mini, and gpt-4o, with the same RAG pipeline are evaluated on the same PubMed dataset across three different performance metrics Mean Reciprocal Rank (MRR), Faithfulness, and Semantic Answer Similarity (SAS). All three models scored a perfect MRR score, meaning good document retrieval but the reasoning models performed better than non-reasoning baseline in faithfulness and SAS. O1-mini model recorded the optimal somatic quality versus roundedness tradeoff. The outcome indicates that it is possible to improve the quality and credibility of produced answers by using the reasoning-capable LLMs for test generation in RAG systems.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.