Search Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses
Abstract
Side-by-side comparisons that elicit human preferences are commonly used to assess the quality of large language model (LLM) output and have been applied to retrieval-augmented generation (RAG) systems. However, when applied to complex, information-seeking queries, these methods are limited by their inability to provide explanatory or diagnostic insights. As an alternative, nugget-based evaluations that decompose long-form answers into atomic facts have emerged as a promising strategy for RAG evaluation. In this work, we employ AutoNuggetizer, a nugget-based framework, to analyze ?5K Search Arena battles from LMArena by automatically generating and assigning nuggets, converting each model response into a quantitative score. We observe strong alignment between nugget-based Elo rankings and human preferences, exceeding the corresponding alignment achieved by LLM-as-a-judge with chain-of- thought (CoT) evaluation, while substantially reducing the number of preference inversions. Furthermore, we provide in-depth analyses including inversions, nugget quality, and shared-blindness effects. All our code is available at https://github.com/castorini/SxSNuggets.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.