Nathan Boudol

Rating the Raters: Rasch Measurement Theory for LLM Evaluation

LLMs now sit on every side of evaluation: as examinees scored on benchmarks, judges of other models'outputs, and raters of human-generated content. Each paradigm can be viewed as a measurement problem, where a latent property of an object is probed with items from an instrument (e.g., benchmark) by judges or raters. St...