Search Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses

Abstract

Side-by-side comparisons that elicit human preferences are commonly used to assess the quality of large language model (LLM) output and have been applied to retrieval-augmented generation (RAG) systems. However, when applied to complex, information-seeking queries, these methods are limited by their inability to provide explanatory or diagnostic insights. As an alternative, nugget-based evaluations that decompose long-form answers into atomic facts have emerged as a promising strategy for RAG evaluation. In this work, we employ AutoNuggetizer, a nugget-based framework, to analyze ?5K Search Arena battles from LMArena by automatically generating and assigning nuggets, converting each model response into a quantitative score. We observe strong alignment between nugget-based Elo rankings and human preferences, exceeding the corresponding alignment achieved by LLM-as-a-judge with chain-of- thought (CoT) evaluation, while substantially reducing the number of preference inversions. Furthermore, we provide in-depth analyses including inversions, nugget quality, and shared-blindness effects. All our code is available at https://github.com/castorini/SxSNuggets.