JudgeBreak, a benchmark of 95 evaluation settings spanning creative writing, summarization, safety, safety, 10 languages, and 33 writing genres, is introduced, showing that text quality is often more recoverable.
Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different...
This work studies whether smaller language models can serve as efficient and reliable rubric-based judges, and compares three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges.
Feng-Yu Xie, Yilun Zhao, Bingsen Chen et al.· 0 citations
This work treats LLM judges as raters and runs a pre-registered rater-effects battery on public corpora in two languages, finding no credible evidence that judge halo exceeds the trained-human range.
An open contrastive decision model is near chance as a judge on the hard public benchmarks: Contrastive-LM/CLM-v0.1-8B scores between 0.351 (best- of-four, chance 0.250) and 0.593 (pairwise, chance 0.500), is statistically indistinguishable from coin flipping on RM-Bench and JudgeBench, and answers every HaluEval item...
LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free...
Daeho Lee, Mihai Delgeanu, Adel Youssef· 0 citations
A wrong verdict does not mean the judge lacks the information, and a simple diagnostic shows when it is worth recovering, and on rubric tasks that score one answer at a time, reading the internals gives no advantage.
