The way advisers split provides a common basis for predictive multiplicity, reconciliation load, and reliance miscalibration and separate aggregation from disclosure and turn the latter into testable questions about how disagreement should be presented and interpreted.
Abstract
When the same question is asked of multiple AI advisers, as in self-consistency and LLM-as-a-judge panels, Condorcet's jury theorem predicts that adding independent, competent advisers makes the majority more reliable. The theorem, however, has a latent dimension when viewed from the user's vantage: adding advisers also makes disagreement more visible. A binomial model reveals that this ``visible dissent''becomes nearly inevitable as the number of advisers grows, and that reliability and disagreement approach certainty at rates that cross at an adviser accuracy of 4/5 (0.8); below it, visible dissent eventually becomes more likely than a correct majority. Even ideal panels of independent and competent advisers can be correct in aggregate but appear divided; such disagreement does not by itself indicate aggregation failure. The way advisers split also provides a common basis for predictive multiplicity, reconciliation load, and reliance miscalibration. These results separate aggregation from disclosure and turn the latter into testable questions about how disagreement should be presented and interpreted.
The jury, with nine experts of up to 3B parameters, performs comparably to the same fusion rule with label-fitted weights, showing no statistically significant advantage from supervision, and outperforms the best single model with matched self-consistency and the strongest label-free alternative on the full benchmark.
A. Rozza, D. Mambelli, S. Motta et al.· 0 citations
Combining multiple imperfect advisors via majority vote is a standard way to improve decision reliability, but real advisors’ errors are correlated, which caps achievable accuracy and causes the benefit of additional advisors to saturate. Existing theory, effective-sample-size (ESS) approximations and correlated Condor...
Ahsan Habib Shihab, Aman Mussa, M. Mansurova· IEEE International Conferenc...· 0 citations
Content moderation compresses a graded, contested human judgment into a binary act (leave up / take down) over billions of items, at a scale where running a frontier LLM on every post is uneconomical. A new class of frozen"System One"models returns typed, calibrated decisions instead of text at ~USD 0.000025 per call a...
Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwise design with 9,312 judgments. A common per-family statistic is strongly confoun...
David Ababio Awuni, Luke Achenie, Benjamin Tei Partey et al.· arXiv.org· 2 citations
Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model...
Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli et al.· 0 citations
LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings; in this paper, we focus on absolute scoring tasks, which mirror more realistic use case...
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.
MIT News · Artificial Intelligence· news.mit.eduJul 7, 2026
The professor of physics and inaugural director of the NSF AI Institute for Artificial Intelligence and Fundamental Interactions will lead LNS and continue his research in particle physics.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.