Hitl-RAG faithfulness evaluation dataset for cultural tourism information
Abstract
This dataset supports accuracy evaluation for Retrieval-Augmented Generation (RAG) systems applied to cultural tourism information, with a specific focus on Human-in-the-Loop (HITL) governance. Designed as an evaluation artifact rather than a performance benchmark, the dataset emphasizes traceability, foundational accuracy, and decision-making transparency. It comprises structured query-response evaluations, including retrieved text snippets, candidate answers generated by RAG-based language models, reference annotations from LLM-based evaluators, automated similarity metrics, and final human evaluation decisions. Accuracy is assessed against curated cultural knowledge snippets, while system-level fallback mechanisms and failure handling are explicitly distinguished from content-level fundamentals. A structured HITL error taxonomy (E0–E10) is applied to differentiate between accurate responses, controlled refusals, content errors, and system failures. Repeated queries are intentionally included to observe consistency across evaluation cycles, enabling analysis of generative variation while maintaining the stability of classification results based on human assessment. Version management follows a snapshot-based policy to ensure reproducibility and cross-version comparability. This release (Version 1.3) contains no substantial changes other than improved image quality and the addition of metadata tables. The inclusion of technical specifications for the LLM models used…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.