Task-Oriented RAG Evaluation: Question-Answer-Citation Benchmark Dataset
Abstract
This repository contains a Question-Answer-Citation (QAC) benchmark dataset developed for task-oriented evaluation of retrieval-augmented generation (RAG) systems. The dataset is designed to evaluate RAG behavior across heterogeneous information-seeking tasks rather than treating all questions as equivalent evaluation instances. Each QAC record connects four elements: 1. a task category representing the underlying information need;2. multiple meaning-preserving variants of the same question;3. an author-derived reference answer; and4. one or more citations identifying the authoritative source evidence supporting that answer. The dataset is constructed from publicly available U.S. Department of the Air Force policy and procedural documents. It supports controlled evaluation of retrieval, context construction, generation, grounding, attribution, and refusal behavior across ten task categories. The ten categories are: 1. Reference / Navigational2. Factoid3. Definition / Conceptual4. Summarization5. Procedural / Task-Oriented6. Policy Interpretation / Compliance7. Comparative / Cross-Document8. Analytical / Rationale9. Ambiguous / Exploratory10. Out-of-Scope / Refusal-Required The classification framework is described in greater detail in taxonomy.md. --- Repository Contents ├── README.md├── QAC_full.json├── qac-schema.json├── taxonomy.md└── docs/ ├── afi1-1.pdf ├── afi36-2113.pdf ├── afi36-2502.pdf ├── afi36-2504.pdf ├── dafi36-2110.pdf ├── dafi36-2501.pdf ├── dafi36-2606.pdf…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.