Skip to content
Preprint

Science Prediction Universe

Nov 2026 · 0 citations
Computer Science

Abstract

Humanity has long sought to explain natural phenomena. Reliable prediction of experimental outcomes is a central test of our understanding of the processes that produce them. If artificial general intelligence (AGI) is to surpass human scientific capabilities, it must also predict experimental outcomes more reliably than humans. Such capabilities could accelerate scientific discovery across disciplines, help address humanity's most pressing challenges, and improve human well-being. We introduce Science Prediction Universe (SPU), a benchmark of hidden experimental outcomes with 100 public verified tasks from chemistry, materials science, biology and medicine, and 16 private tasks from unpublished wet-lab experiments. Given an experimental setup and candidate conditions, an agent works with any resources it needs to rank the candidates by a measured outcome. We evaluate thirteen agent configurations spanning twelve frontier models across two harnesses (Codex and Claude Code). Current agents remain far from saturating wet-lab outcome prediction, with substantial performance disparities across models: on the public benchmark, configurations separate into two distinct tiers, with task-adjusted Spearman correlations ranging from 0.227 for the lowest-performing configuration to 0.478 for the best. Analysis of 1,155 scored public trajectories (227,103 tool calls) suggests that the bottleneck lies in scientific judgment rather than tool execution: agents rely on shared monotone priors, agreeing far more with one another than with experimental reality (0.52 vs. 0.33), and frequently compute trivial physical or chemical proxies that do not influence the final outcome.

View source

Similar papers

Open access Sep 2026

GEMOT: Towards Mechanistic World Models for Biology

Scientific discovery seeks mechanisms that explain observations and predict beyond the measurements that produced them. Whereas large language models (LLMs) encode knowledge implicitly, mechanistic world models organise it as a parsimonious set of explicit, modular, reusable mechanisms whose predictions can be scored a...

Theodore de Pomereu, B. Snelling, Fabian Fröhlich · 2 citations
Preprint Nov 2026

The directions of science: How we search the unknown

Science is producing knowledge at unprecedented scale, while artificial intelligence promises to accelerate the generation of hypotheses, analyses, and scientific outputs further. Yet increasing the volume of research does not necessarily expand the range of directions science effectively explores. We argue that scienc...

Nicola Cinardi · 0 citations

Can AI Scientists Change Their Minds? Prior-Evidence Conflict in Synthetic Universes

Synthetic Universes is introduced, a controlled benchmark that pairs canonical famous worlds with matched twisted twins governed by nearby noncanonical mechanisms and establishes a narrower verification result: predictive adequacy and mechanism recovery are distinct scientific claims and require distinct tests.

Kargi Chauhan · 0 citations
#artificial intelligence Preprint Oct 2026

From Scientific Observations to Mechanisms: Benchmarking Hypothesis Generation by AI Scientists

MechHypoBench is introduced, the first benchmark for evaluating whether AI agents and AI scientists can generate mechanistic hypotheses from empirical data, and an evaluation framework is developed that assesses open-form mechanistic hypotheses through their consequences under withheld conditions.

Xia-Xun Xie, Qing-Qing Long, Meng Xiao et al. · 0 citations

Artificial intelligences and human scientists exhibit complementary strengths in theory building

We investigate the effectiveness of artificial intelligences (AI)-specifically large language models (LLMs)-relative to human scientists at high-level cognitive tasks in social science such as theory formulation, predictions of novel empirical results, and theory revision in response to new evidence. The research domai...

Ke Li, S. Zoumpoulis, P. Puranam et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.