Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks
Most multiple-choice question (MCQ) benchmarks evaluate Large Language Models (LLMs) only by whether they select the correct answers. This binary scoring treats all incorrect responses alike, even though an LLM's preferences among incorrect options may contain systematic and useful information about its behavior and ab...