Parameter Optimization for Retrieval-Augmented Generation Systems
Abstract
Retrieval-Augmented Generation (RAG) is a modeling approach that incorporates domainspecific context into Large Language Models (LLMs) in order to provide more accurate answers to user queries. This thesis investigates how answers generated by RAG models can be effectively evaluated using suitable metrics and identifies optimal context-related parameters for such models. The large language models used in this work are based on the transformer architecture, which predicts the most probable tokens using attention mechanisms. Retrieval-Augmented Generation employs hybrid search, combining semantic search with traditional keyword-based search to retrieve the most relevant segments of the context, referred to as chunks. Using the retrieved context, predefined instructions, and the user query, the RAG model generates a response. The generated answers are evaluated using the RAGAS library, which provides multiple metrics for RAG validation. In this thesis, answer similarity, answer correctness, context precision, and context recall are used to assess overall accuracy and contextual relevance. Some of these metrics require a ground truth answer, which serves as a reference for comparison with the modelgenerated response. The ground truth is constructed by a domain specialist to ensure correctness and contextual appropriateness. Each metric evaluates a different aspect of the RAG pipeline, and together they provide a reliable validation framework. Bayesian optimization techniques are…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.