Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena
Chatbot Arena ranks language models with the Bradley-Terry model, which gives each model one score and so assumes Luce's choice axiom. Choice theory predicts where this fails: near-copies are told apart more decisively than their scores imply. On 1.8 million Arena votes, votes between versions of one model are 1.37 tim...
S. Bakshi, Apurba Bose, Aishwarya Bose et al.
· 0 citations