When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
It is shown that raw global calibration metrics are not robust for cross-model comparison, and that fair calibration comparison requires accuracy-aware evaluation, and proposed ACE, an accuracy-controlled evaluation framework, is proposed.