Evaluating Modular Retrieval-Augmented Generation : A Case Study on the Sprakradet Language Advice Corpus
Abstract
This thesis examines the integration of advanced Retrieval-Augmented Generation (RAG) modules within institutional settings demanding strict source-bounded accuracy. The primary research question addresses whether standalone advanced modules and a deterministic orchestrated pipeline improve retrieval metrics, context purity, and query refusal behavior compared to a basic RAG baseline. The methodology evaluates Corrective RAG (CRAG), Query Decomposition, and GraphRAG across standardized benchmarks (BIRCO) and a specialized Swedish language-advice corpus (Språkrådet), using an LLM-as-a-judge framework for end-to-end generative assessment. Results demonstrate that while high-dimensional dense embeddings establish a strong performance baseline, the CRAG module yields the most pronounced utility by systematically filtering out noise and raising context relevance. Conversely, topological GraphRAG underperforms in exact-passage extraction tasks. While orchestration does not produce a transformative leap in generative accuracy over individual standalone sub-components, it provides an auditable, step-by-step verification framework that achieves a 95% refusal rate on unanswerable queries. These findings indicate that structured context-filtering layers offer the necessary transparency and verification guardrails required for high-consequence institutional deployments.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.