RheumBench: A Specialist-Validated Rubric-Based Benchmark for Evaluating Large Language Models in Rheumatology
Abstract
Abstract Large language models (LLMs) are increasingly used for clinical decision support, yet scalable and specialty-specific methods for evaluating their performance and safety remain limited. We developed RheumBench, the first rubric-based LLM benchmark in rheumatology, providing a specialist-validated framework for open-ended evaluation. We evaluated ten systems, comprising three frontier proprietary general-purpose LLMs, one frontier open-weight general-purpose LLM, and six clinical retrieval-augmented generation (RAG) systems, including two certified medical products and one rheumatology-specific RAG system. RheumBench comprised 100 clinical cases (70 diagnostic and 30 diverse clinical management) and 1,020 case-specific rubric criteria independently reviewed and weighted for clinical appropriateness and potential harm by two board-certified rheumatologists. Systems were evaluated for answer quality, prompt sensitivity, potential harm, diagnostic accuracy, and system confidence. Responses were scored by a three LLM judge panel, yielding 50,553 rubric-level LLM evaluations, with high agreement with rheumatologist consensus (Cohen’s κ = 0.91). Overall system performance (0-100%) varied substantially, with a maximum RheumBench score of 54.0% (Claude Fable 5), and frontier general-purpose LLMs generally outperformed clinical RAG systems, including certified Prof. Valmed and DR.INFO. Management case performance varied widely, with thorough prompting improving mean…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.