Subset Heterogeneity in Reward Model Benchmarks: A CONFIRM Validation of 174 RewardBench 2 Models
Background. RewardBench 2 aggregates pairwise preference accuracy across heterogeneous task families (factuality, instruction following, safety, and others). A single leaderboard score can mask systematic subset specialization, yet standard benchmark reporting rarely tests whether accuracy is independent of task catego...