Humanity has long sought to explain natural phenomena. Reliable prediction of experimental outcomes is a central test of our understanding of the processes that produce them. If artificial general intelligence (AGI) is to surpass human scientific capabilities, it must also predict experimental outcomes more reliably than humans. Such capabilities could accelerate scientific discovery across disciplines, help address humanity's most pressing challenges, and improve human well-being. We introduce Science Prediction Universe (SPU), a benchmark of hidden experimental outcomes with 100 public verified tasks from chemistry, materials science, biology and medicine, and 16 private tasks from unpublished wet-lab experiments. Given an experimental setup and candidate conditions, an agent works with any resources it needs to rank the candidates by a measured outcome. We evaluate thirteen agent configurations spanning twelve frontier models across two harnesses (Codex and Claude Code). Current agents remain far from saturating wet-lab outcome prediction, with substantial performance disparities across models: on the public benchmark, configurations separate into two distinct tiers, with task-adjusted Spearman correlations ranging from 0.227 for the lowest-performing configuration to 0.478 for the best. Analysis of 1,155 scored public trajectories (227,103 tool calls) suggests that the bottleneck lies in scientific judgment rather than tool execution: agents rely on shared monotone priors, agreeing far more with one another than with experimental reality (0.52 vs. 0.33), and frequently compute trivial physical or chemical proxies that do not influence the final outcome.
EDB-Bio is introduced, a scientific reasoning benchmark curated from real-world data in three biological domains: drug-perturbation transcriptomics, CRISPR screens, and human genome-wide association studies (GWAS).
C. Igoe, Gabriel Grand, Zane Koch et al.· 0 citations
Scientific discovery seeks mechanisms that explain observations and predict beyond the measurements that produced them. Whereas large language models (LLMs) encode knowledge implicitly, mechanistic world models organise it as a parsimonious set of explicit, modular, reusable mechanisms whose predictions can be scored a...
Theodore de Pomereu, B. Snelling, Fabian Fröhlich· bioRxiv· 2 citations
Science is producing knowledge at unprecedented scale, while artificial intelligence promises to accelerate the generation of hypotheses, analyses, and scientific outputs further. Yet increasing the volume of research does not necessarily expand the range of directions science effectively explores. We argue that scienc...
Synthetic Universes is introduced, a controlled benchmark that pairs canonical famous worlds with matched twisted twins governed by nearby noncanonical mechanisms and establishes a narrower verification result: predictive adequacy and mechanism recovery are distinct scientific claims and require distinct tests.
MechHypoBench is introduced, the first benchmark for evaluating whether AI agents and AI scientists can generate mechanistic hypotheses from empirical data, and an evaluation framework is developed that assesses open-form mechanistic hypotheses through their consequences under withheld conditions.
Xia-Xun Xie, Qing-Qing Long, Meng Xiao et al.· 0 citations
We investigate the effectiveness of artificial intelligences (AI)-specifically large language models (LLMs)-relative to human scientists at high-level cognitive tasks in social science such as theory formulation, predictions of novel empirical results, and theory revision in response to new evidence. The research domai...
Ke Li, S. Zoumpoulis, P. Puranam et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.