Optimizing and Evaluating a Retrieval-Augmented Generation System for Normative Document Retrieval in Hospital Settings
Abstract
This paper presents the development and evaluation of a chatbot designed to consult documents written in Portuguese on regulatory procedures in a hospital environment, which uses a Retrieval-Augmented Generation (RAG) pipeline to increase the factual accuracy and relevance of its underlying Large Language Model (LLM). Using the RAG technique will allow for more efficient and accurate retrieval of information contained in hospital manuals and institutional documents, helping workers quickly access internal guidelines and procedures. The objective is to optimize each system component (retrieval, re-ranking, and generation) to analyze the impact of each step in developing a RAG system for a low-resource language such as Portuguese. The methodology can be divided into the following stages: (1) the Golden Set Preparation, formed by a set of questions and answer data; (2) comparison of three embedding models for initial retrieval and of three re-ranking methods, including Cross-Encoder, Reciprocal Rank Fusion (RRF), and an LLM-based re-ranker, using metrics such as MRR, NDCG@10; and (3) comparison of two generative models (Gemini 1.5 Flash and GPT-4o-mini), using the metric BERTScore. The results indicate that the intfloat/multilingual-e5-small embedding model minimizes retrieval failures. In the re-ranking stage, the LLM-based re-ranker achieved the highest ranking accuracy, yet the computationally lighter RRF method emerged as an excellent cost-effective alternative. We conclude…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.