Fairness trade-offs in expert evaluation: information, judgment, and procedural design

Abstract

Evaluation systems often seek to reduce bias, prevent manipulation, preserve relevant information, maintain comparability, and treat candidates equally, but these fairness goals can conflict. This paper develops a framework for understanding such trade-offs in expert evaluation—including peer review, grant panels, academic hiring, and artistic competitions—where quality is multidimensional and legitimate disagreement among evaluators is possible. We treat the choice among evaluation procedures as a problem of constitutional design: rules must be selected before the specific candidates, evaluators, and circumstances that will test them are known. Classical music competitions are our primary analytical setting because their rules, scores, and controversies make these trade-offs unusually visible, but the framework applies to other settings in which expert judgments are aggregated under incomplete information. The framework highlights two core tensions: information control and outlier rules. First, when information is both quality-relevant and identity-revealing, procedures that suppress bias-relevant cues may also remove information with genuine evaluative content. Second, because sincere minority judgment and strategic manipulation can generate similar observed score patterns, score-based exclusion rules may reduce manipulation while limiting legitimate dissent. Two further trade-offs concerning transparency and standardization are developed as illustrations. The analysis shows that no procedure can eliminate all vulnerabilities at once. Beyond this design implication, the framework also helps explain why evaluation rules differ across stages and settings, and why well-intentioned reforms may create new vulnerabilities.