RAGLens: a study on understanding, evaluating, and comparing retrieval-augmented generation techniques (2nd Gen)
Abstract
Retrieval-Augmented Generation (RAG) has become a core building block of modern large language model applications, yet there is no one-size-fits-all solution, the effectiveness of different pipeline configurations varies across datasets, domains, and query types. Most existing research focuses on introducing new techniques and reporting benchmark performance, leaving a gap in practical tooling that supports developers in navigating, comparing, and understanding these choices. This project presents RAGLens (Gen-2), a cloud-deployed, developer-centred platform for exploring, comparing, and evaluating RAG pipelines. Building on the first-generation prototype, this second-generation system addresses three key limitations: limited accessibility, the absence of formal user evaluation, and insufficient support for emerging techniques. RAGLens (Gen-2) supports nine RAG techniques across pre-retrieval, retrieval, and post-retrieval stages, providing a learning-oriented interface with technique-specific explorer pages, step-by-step execution traces, side-by-side comparison with automated G-Eval scoring and Meta-LLM recommendation, and benchmark evaluation using RAGAS. To ground the system in real developer needs, a structured developer survey was conducted with 25 respondents spanning varying roles and experience levels. Findings revealed that current RAG development practices are largely ad hoc, evaluation confidence is low, and developers seek tools that support interpretability,…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.