How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats
To help researchers understand how to run statistics over LLM judges responsibly, guidance and tooling for the statistical analysis of mixed human-AI judge designs are presented, and recommendations for what CI, p-value, and FWER correction methods to use for small-sample AI evaluations are derived, and researchers are...