TrustRAG: evidence localisation and failure attribution for retrieval-augmented generation evaluation (v1.0.0)
Abstract
Code, per-question result files and reproduction commands for the measurement study *Right document, wrong passage: evidence localisation in retrieval-augmented generation evaluation on QASPER and Natural Questions*. The release contains the corpus loaders and chunker (QASPER and Natural Questions are fetched from their original sources and are not redistributed), the within-document localisation protocol with its analytic chance level, the seven-localiser comparison (BM25, four dense bi-encoders, two cross-encoder rerankers) with Holm-corrected paired tests and a document-level cluster bootstrap, the union, rank-correlation, admission-budget and chunk-size analyses, the corpus-contrast script, the annotation interface and audit scripts behind the human-reviewed failure-attribution study and the gold-span completeness adjudication, the oracle-evidence experiment, and the test suite. The human annotation package (200 QASPER units with retrieved context, original/review/final labels with provenance, guideline and truncation audits, gold-span adjudication sheet and answers, threshold-ablation and oracle-evidence rows) is included as a separate archive associated with this record. QASPER-derived materials are distributed under the redistribution terms of QASPER (CC BY 4.0).
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.