ContextCoherenceBench: a benchmark for evaluating context coherence in retrieval-augmented generation
Abstract
Companion benchmark for the paper "When Better Retrieval Hurts: Context Coherence Drives Faithfulness in Retrieval-Augmented Generation" (NeurIPS 2026 submission). Contains: Adversarial coherence test set (~129 validated cases across four categories: drift, disjoint, contradict, control). Coherence-paradox test pairs across SQuAD, PubMedQA, HotpotQA, NaturalQuestions, and TriviaQA, evaluated against three open-weight generators (Mistral-7B, Llama-3-8B, Qwen-2.5-7B) and three retrieval configurations (baseline, HCPC-v1, HCPC-v2). Context Coherence Score (CCS) computed per query, alongside per-passage similarity and faithfulness (NLI) judgments. Frontier-scale rows from Llama-3.3-70B and GPT-OSS-120B via Groq, demonstrating the paradox persists at scale. Leaderboard schema for community submissions (leaderboard.yaml) with rules and submission template. The bundle includes a CITATION.bib entry, an MIT-licensed LICENSE file, a HuggingFace-loadable layout, and a SHA-256 manifest for every shipped file (see metadata.json). Reproducibility code is at https://github.com/Saket-Maganti/rag-hallucination-detection; an interactive Gradio demo is at https://huggingface.co/spaces/saketmgnt/sakkk.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.