Public AI Benchmarks Are Broken, But Are Private Benchmarks the Answer?

Sang T. Truong S. Wang Angelina Wang Nhi N. Truong Anka Reuel Nick Haber Sanmi Koyejo Stanford Cornell Harvard

· 0 citations · 38 references

TL;DR

This work argues for a hybrid public-private approach: keeping the benchmark content private to maintain integrity while making the benchmark design public to build trust, which offers a path toward trustworthy measurement of ever-growing data-intensive AI systems.

Semantic Scholar

View source

Similar papers

Review Aug 2026

Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation

A blockchain-based commit-reveal protocol using Autonomous Economic Agents on an Ethereum-compatible ledger creates a tamper-evident audit trail that separates blind evaluation from post-hoc claims and reduces the verification burden on independent researchers and leaderboard operators.

Sahil Pardasani, Madhusudan Singh · 0 citations

Preprint Aug 2026

FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents

Evaluating FrontierFinance, a fully open benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow, finds that the tool harness, not the model alone, strongly shapes quality and efficiency.

Yuhao Zhang, O. Koyluoglu, Thejas Venkatesh et al. · 3 citations · ⚡2

Preprint Sep 2026

What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks

This work adapt convergent and discriminant validity from the social sciences into an approach for interrogating AI benchmarks, applying it to 56 capability and safety benchmarks across 53 models and asking analogous questions at the item level using item response theory (IRT) models.

Meera Desai, Sang T. Truong, Hanna Wallach et al. · 0 citations

#machine learning Preprint Aug 2026

One Capability or Many? Testing the Economic Validity of Frontier AI Evaluation

Economic benchmarks add incremental predictive information to a largely date-driven general factor, and the evidence does not support treating them as a distinct latent capability.

L. Zhu · 0 citations

Preprint Sep 2026

The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean

The results suggest that benchmark scores should be interpreted together with their scaffolding level, scoring criterion, and reliability profile, providing a practical framework for more valid evaluation of LLM agents.

Yong-Hong Zhang, Shadi Motaali, Vu Phong Dinh et al. · 3 citations

Jul 2026

A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models

A consensus-based evaluation framework that measures relative preference among model-generated responses rather than absolute correctness rather than absolute correctness is introduced, offering an alternative perspective on response quality in scenarios where multiple valid answers exist.

Mohtashim Khan · arXiv.org · 0 citations