RAGtio: A Modular Framework for Systematic Evaluation of Hybrid Retrieval-Augmented Generation Pipelines
Abstract
Abstract Retrieval-Augmented Generation (RAG) architectures combine Large Language Models (LLMs) withexternal knowledge stores to produce grounded, verifiable answers. While RAG systems have beenincreasingly adopted in the biomedical domain, existing work focuses primarily on downstream question-answering (QA) performance, with limited attention to systematic evaluation and usability for non-technical users. This paper presents RAGtio, a modular, open-source framework for the rigorous andreproducible evaluation of RAG retrieval pipelines over domain-specific text corpora, with primary appli-cation to biomedical documents. The framework implements and allows the comparison of four retrievalconfigurations: sparse, dense, hybrid, and hybrid with cross-encoder re-ranking, using a weighted Re-ciprocal Rank Fusion (RRF) mechanism. A key contribution is the introduction of two complementaryevaluation modes: (i) Mode A automatically generates synthetic queries from randomly sampled docu-ment chunks via an LLM, enabling rapid large-scale assessment; (ii) Mode B accepts user-provided queryrelevance pairs assembled from domain expertise, yielding an unbiased estimate of retrieval quality ongenuinely novel queries. The framework is built on the Haystack library and Qdrant vector database,supports heterogeneous document formats, and exposes a YAML-based configuration interface togetherwith a web-based graphical interface for running evaluations and inspecting results without…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.