Evaluation of a Retrieval-Augmented AI Audit Assistant for Financial Document Compliance Screening and Accounting Anomaly Detection
Abstract
Objective. Structured numeric review and unstructured textual audit evidence remain handled by largely disconnected systems, leaving a persistent gap between flagging an anomalous transaction and grounding that finding in applicable business or regulatory text. This study developed and evaluated a retrieval augmented AI audit assistant organized around a five stage Detect, Link, Retrieve, Verify, Explain pipeline, designed to integrate structured anomaly detection with citation grounded evidentiary reasoning. Methods. A three phase, sequential comparative design was adopted, escalating from entry level detection across 10,842 journal entries, to evidence level retrieval across 512 gold annotated question evidence pairs drawn from a 154 document corpus, to case level evaluation across 100 objective and 20 expert blind reviewed audit cases. Four classifiers were compared under traditional, embedding, and fused feature configurations, four retrieval strategies were compared, and four end to end system configurations were assessed against reference audit opinions and blinded expert ratings. Results. Feature fusion improved detection performance across most classifiers, hybrid retrieval with reranking achieved the strongest evidence grounding, and the fully integrated system reached 84.0 percent objective accuracy with a 9.0 percent hallucination rate, outperforming all ablated configurations while reducing review time by roughly 54 percent. Conclusion. Integrating structured…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.