Skip to content

A latent dimension of Condorcet's jury theorem for multiple AI advisers

Sep 2026 · 0 citations · 66 references
Computer Science

TL;DR

The way advisers split provides a common basis for predictive multiplicity, reconciliation load, and reliance miscalibration and separate aggregation from disclosure and turn the latter into testable questions about how disagreement should be presented and interpreted.

Abstract

When the same question is asked of multiple AI advisers, as in self-consistency and LLM-as-a-judge panels, Condorcet's jury theorem predicts that adding independent, competent advisers makes the majority more reliable. The theorem, however, has a latent dimension when viewed from the user's vantage: adding advisers also makes disagreement more visible. A binomial model reveals that this ``visible dissent''becomes nearly inevitable as the number of advisers grows, and that reliability and disagreement approach certainty at rates that cross at an adviser accuracy of 4/5 (0.8); below it, visible dissent eventually becomes more likely than a correct majority. Even ideal panels of independent and competent advisers can be correct in aggregate but appear divided; such disagreement does not by itself indicate aggregation failure. The way advisers split also provides a common basis for predictive multiplicity, reconciliation load, and reliance miscalibration. These results separate aggregation from disclosure and turn the latter into testable questions about how disagreement should be presented and interpreted.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

A Jury, Not an Electorate: Label-Free Reliability Estimation for Ensembles of Language Models

The jury, with nine experts of up to 3B parameters, performs comparably to the same fusion rule with label-fitted weights, showing no statistically significant advantage from supervision, and outperforms the best single model with matched self-consistency and the strongest label-free alternative on the full benchmark.

A. Rozza, D. Mambelli, S. Motta et al. · 0 citations
Conference Sep 2026

How Many Advisors Are Enough? Predicting Useful Majority-Vote Ensemble Size under Correlated Decision Errors

Combining multiple imperfect advisors via majority vote is a standard way to improve decision reliability, but real advisors’ errors are correlated, which caps achievable accuracy and causes the benefit of additional advisors to saturate. Existing theory, effective-sample-size (ESS) approximations and correlated Condor...

Ahsan Habib Shihab, Aman Mussa, M. Mansurova · 0 citations
#natural language process... Preprint Oct 2026

Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

Content moderation compresses a graded, contested human judgment into a binary act (leave up / take down) over billions of items, at a scale where running a frontier LLM on every post is uneconomical. A new class of frozen"System One"models returns typed, calibrated decisions instead of text at ~USD 0.000025 per call a...

Karan Yadav · 0 citations

Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels

Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwise design with 9,312 judgments. A common per-family statistic is strongly confoun...

David Ababio Awuni, Luke Achenie, Benjamin Tei Partey et al. · 2 citations
#artificial intelligence Preprint Oct 2026

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model...

Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli et al. · 0 citations

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings; in this paper, we focus on absolute scoring tasks, which mirror more realistic use case...

Gemma Zhang, Prachi Badarayani, Asmi Kumar et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.