Assessing AI and Neurologist Diagnostic Reasoning Against Neuropathological Ground Truth
Abstract
BACKGROUND: Accurate differential diagnosis of complex neurological disorders remains challenging due to overlapping clinical features and heterogeneous disease presentations. Although large language models (LLMs) show promise in clinical reasoning, prior studies benchmark performance against clinician consensus rather than biological ground truth. A neuropathologically confirmed benchmark dataset for evaluating diagnostic AI in neurology is currently lacking.
METHODS: We introduce NeuroBench, a curated benchmark of complex neurological cases with neuropathologically confirmed gold-standard diagnoses, and DIAGNO, a confidence-aware LLM-based system for neurological diagnosis. NeuroBench comprises 203 retrospective case summaries from the Massachusetts General Hospital Brain Cutting Conference with corresponding autopsy-confirmed diagnoses. DIAGNO generated top-3 differential diagnoses, employing retrieval-augmented generation (RAG) for lower-confidence cases. Performance was assessed by three independent blinded adjudicators who evaluated both DIAGNO and neurologists against neuropathological ground truth.
RESULTS: NeuroBench encompassed 79 unique neuropathological diagnoses, spanning conditions including cerebrovascular disease, brain tumors, neurological infections, and various neurodegenerative and inflammatory disorders. DIAGNO matched or outperformed neurologists in top-3 accuracy (0.67 versus 0.63) and taxonomy-level accuracy (0.74 versus 0.66). In cases of…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.