SUS LLM Hallucination Dataset: A curated benchmark for evaluating hallucinations in large language models in a regulated public health system
Abstract
This dataset provides a curated benchmark for evaluating hallucinations in large language models (LLMs) in a highly regulated public health system. The dataset contains 90 structured questions covering key domains of Brazil’s Unified Health System (SUS), including coverage, care pathways, eligibility criteria, and patient rights. Each item is annotated with a taxonomy of hallucination types (T1–T6), risk levels, and associated normative sources, enabling systematic evaluation of factuality, compliance, and safety in LLM-generated responses. The dataset is designed for use in:- Benchmarking LLM performance in regulated environments- Evaluating retrieval-augmented generation (RAG) systems- Research in AI safety, governance, and hallucination mitigation- Public sector AI evaluation and policy-oriented research This resource emphasizes high-risk, regulation-sensitive scenarios, where hallucinations may have real-world consequences. All data are provided in structured formats with accompanying documentation to support reproducibility and reuse.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.