Leveraging Collective Advice-Taking Behavior to Infer Accuracy and Improve the Wisdom of Crowds
TL;DR
Cluster weight on advice (CWOA) is proposed, a novel “two-shot” algorithm to identify a more accurate subgroup in a single-prediction-problem context and consistently outperforms benchmarks, including the state-of-the-art metaprediction-based methods.
Abstract
Wisdom of crowds estimates can be compromised when some agents’ predictions are systematically biased. A natural remedy is to aggregate predictions from a subset of more accurate agents rather than the entire crowd. I propose cluster weight on advice (CWOA), a novel “two-shot” algorithm to identify a more accurate subgroup in a single-prediction-problem context. CWOA first applies kernel density estimation to identify clusters of similar initial predictions. If multiple clusters emerge—indicating potential heterogeneity in agents’ information—the algorithm proceeds to present a piece of numerical advice (e.g., the group mean) and elicit updated predictions. This enables the calculation of the weight on advice (WOA)—the scaled magnitude of each agent’s belief revision. CWOA then averages the updated predictions within the cluster with the lowest mean WOA. A behavioral model and simulations explain both why and when cluster-level WOA signals accuracy. Better-informed agents—having already incorporated higher-quality information—perceive less corrective value in the advice and therefore, exhibit lower WOA. CWOA does not require agents to know the true biases or the composition of the crowd; a modest relative advantage in perceived estimation bias by better-informed agents may be sufficient, even under misperceptions of variance, advice quality, and psychological biases in advice taking. Empirically, I first test and confirm the model’s key insight in a controlled experimental setting. I then validate CWOA’s performance across multiple preregistered and archival data sets, including a study in which numerical advice comes from artificial intelligence. CWOA consistently outperforms benchmarks, including the state-of-the-art metaprediction-based methods. This paper was accepted by Jack Soll, behavioral economics and decision analysis. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.04499 .