Building a Reliable Retrieval-Augmented Generation (RAG) System with Hallucination Awareness
Abstract
Retrieval-augmented generation has become an important approach for improving the factual grounding of large language model outputs by conditioning generation on externally retrieved documents. However, retrieval alone does not guarantee reliable answers. A system may retrieve irrelevant or incomplete evidence, generate unsupported claims, or fail to communicate uncertainty when the available knowledge is insufficient. This thesis investigates how RAG systems can be designed and evaluated with a focus on reliability and hallucination-aware behaviour in a fixed-size knowledge base. The work follows a system-building and evaluation approach. A modular RAG prototype was developed to support controlled experimentation across different pipeline configurations. The system was evaluated on a researchpaper corpus using a custom query dataset containing answerable, ambiguous, and unanswerable questions. Three main retrieval configurations were compared: baseline dense retrieval, query expansion with reranking, and dynamic retrieval based on reranker score gaps. The experiments also compared three embedding models and multiple retrieval depths. The evaluation used retrieval-side metrics, including recall, hit rate, and mean reciprocal rank, together with generation-side metrics such as answer correctness and faithfulness. The results show that retrieval-side design choices substantially affect system reliability. The choice of embedding model had a strong impact on retrieval quality,…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.