Retriever or reasoner? Decomposing retrieval-augmented generation performance in external audit supervision
Abstract
Supervisory reviews of external audit reports require structured evidence extraction and judgement under strict confidentiality and governance constraints. We evaluate retrieval-augmented generation pipelines for this task, comparing lexical, semantic, hybrid, and oracle retrieval across on-premise open-weight models (Llama 3B, Mistral 7B, Llama 70B) and proprietary cloud models (Kimi, Claude Sonnet 4.6). Using 20 bank audit reports and a standardized central bank template of 30 questions, we score operational correctness against supervisor-provided ground truth with an independent LLM-as-judge, complemented by expert back-to-back checks. The paired design holds each question report pair fixed, separating gains driven by retrieval quality from those driven by model capability, and identifying when they operate as complements rather than substitutes. Semantic retrieval yields a sizeable and statistically robust uplift within fixed models. Under symmetric strong retrieval, Llama 70B becomes statistically indistinguishable from the best cloud benchmark, while smaller on-premise models remain constrained on higher complexity judgement questions. The results point to a capacity threshold for practical substitution and provide evidence to guide deployment trade-offs in regulated settings.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.