Scoring-Rule Sensitivity in LLM Evaluation: Evidence from Indian Financial Regulatory Text

Abstract

A benchmark leaderboard reports one accuracy number, but that number is a joint product of a model and a scoring rule. We ask what changes when only the rule changes, and find that model ordering is highly sensitive to it. IndiaFinBench is a 406-item expert-annotated benchmark over SEBI and RBI regulatory text covering regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning. We score twelve LLMs zero-shot under two primary regimes over the same predictions - strict four-stage string matching, and the verdicts of a cross-model judge sharing no model family with any evaluated system - plus one operational composite regime. The two primary regimes show no positive rank correspondence (Spearman rho = -0.273, p = 0.39, not significant), and no model holds its rank across all three regimes. Strict accuracy spans 75.12-89.66%; judge-only accuracy spans 87.19-94.83% with a different leader and trailer; the composite compresses the field to 94.1-98.0%, under which DeepSeek-R1-Distill-Llama-70B ties for first despite ranking last under strict scoring, because the judge reclassifies 93.9% of its strict errors as correct. Output verbosity does not explain the reclassification once task and model composition are adjusted for, a negative result we report as such. Item-level analysis compounds the finding: 148 of 406 items (36.5%) are answered identically by all twelve models and separate no pair of systems, while a predefined higher-discrimination band captures 134 items. We also anchor the scale against one non-specialist evaluator, who scored 80.0% (48/60; 95% Wilson CI [68.2, 88.2]) on a 60-item medium/hard subset; paired bootstrap tests find no significant difference against any model after multiplicity correction. Data, harness, judge verdicts, predictions: https://github.com/Rajveer-code/IndiaFinBench