Skip to content
Preprint

Rewarding Noise: Incentives in Decentralized A/B Testing

Oct 2026 · 0 citations · 18 references
Economics

Abstract

Many firms that run A/B tests struggle with low statistical power, increasing the risk that launch decisions are driven by noise rather than true impact. We argue that in decentralized organizations, low power can reflect misaligned incentives: product teams choose what and how to test on a customer-facing experience to maximize their private reward, while the experience owner sets evaluation rules and launch criteria to optimize the experience. We formalize this interaction via a principal-agent model and show that evaluating tests using raw impact estimates creates a reward to sampling noise. Correctly calibrated posterior-mean evaluation removes this reward and creates a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas. Using data on A/B tests run at Amazon, we document patterns consistent with these model predictions. After an Amazon experience shifted to posterior-mean-based evaluation in 2024, test volume declined while both precision and the dispersion of true impacts increased, more than doubling signal-to-noise ratios. Our results highlight that A/B testing methodologies can shape not only measurement and launch decisions, but which tests are pursued in the first place.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Incentive Alignment in Online Experimentation

Evaluating the causal effect of new features is a central goal for online platforms. While recent literature addresses limited testing traffic via centralized portfolio optimization, this perspective abstracts away a critical institutional reality: experimentation is operationally decentralized. The experimenters who d...

Ermis Soumalias, R. Mudd, Abbas Zaidi · 1 citation
Preprint Oct 2026

World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits

Selecting a reinforcement-learning (RL) policy usually requires additional environment interaction, while learned world models offer cheap but potentially biased evaluations. This raises two distinct questions: can shared model errors preserve pairwise rankings despite poor absolute values, and do they make a small num...

Ahanaf Hasan Ariq, Hasaan Ahmad · 0 citations
Open access Sep 2026

Setting CEO Pay: What Pitfalls Should We Avoid?

benchmarks, variable compensation, and aligning incentives with the company’s stock: this seems like a no-regret move, right? Yet, in far too many cases, CEO pay has not correlated well with the performance of the companies they lead. There are also concerns that their compensation, driven largely by equity, has become...

Anurag Guptai, Allan Zebedeeii · 0 citations
Preprint Oct 2026

What a Shorter Contest Really Costs: Serial Dependence, Effective Trials, and Evidence from a Rule Change in Professional Esports

Operational research evaluates tournament formats by their efficacy, the probability that the design identifies the stronger competitor, and computes it almost exclusively by simulation under an assumption that is rarely tested: that the trials a format aggregates are independent. We show that this assumption systemati...

Petr Parshakov · 0 citations
Preprint Oct 2026

Truthful and Admissible Principal--Agent Testing with Nuisance Parameters

Statistical contracts reward experimenters according to the evidence produced by a costly study, so the reward rule influences both who participates and what they report. Principal--agent testing theory explains how to prevent ineffective proposals from being profitable. We ask when such contracts can also elicit hones...

Aaditya Ramdas · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.