Skip to content

SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning

Oct 2026 · 0 citations · 61 references
Computer Science

TL;DR

The results suggest that SPEEDRUNBENCH is a useful testbed for studying agents's strategy formation capabilities as well as being a saturation-resistant evaluation measure, as there is almost always a faster completion time waiting to be discovered.

Abstract

Frontier LLM agents have been shown to be capable of solving increasingly complex tasks for which humans have measurable solutions. This begs the pertinent question of whether LLM agents can go beyond what humans have already solved. The ability to develop sophisticated strategies to tackle consequential problems becomes paramount as well-trodden, human-developed solutions become insufficient for problems for which we lack context or enough training data. We study agents'capability of such strategy formation through the communal practice of video game speedrunning. In speedrunning, practitioners compete to find the fastest way to complete a video game under certain conditions, and in so doing uncovering interesting unorthodox play styles that require a thorough understanding and mastery of the underlying game mechanics. We introduce SPEEDRUNBENCH, a benchmark that evaluates frontier LLM agents across 9 different games. To perform well in this benchmark, agents must repeatedly improve their strategy, reflect on their performance, exploit their gained knowledge, and reason across a long-horizon of actions to improve on an increasingly difficult problem: being faster than themselves and everyone else. Our experiments show that while frontier agents approach human world records in simple platformer games, they remain behind human performance on longer, more complex games under practical budgets. These results suggest that SPEEDRUNBENCH is a useful testbed for studying agents'strategy formation capabilities as well as being a saturation-resistant evaluation measure, as there is almost always a faster completion time waiting to be discovered.

View source

Similar papers

GameBoyWorlds: A Testbed for Self-Improvement in Embodied Video Games

GameBoyWorlds is introduced, a testbed for agentic self-improvement in video games and it is shown that contemporary approaches to self-improvement are lacking, with world modelling and autonomous skill discovery failing, and a novel strategy that uses curiosity-based exploration to write guides achieving only partial...

Dhananjay Ashok, Adam Shen, Aslan H. Feng et al. · 1 citation
Conference Nov 2026

Generating Board Games with MCTS and LLM Agents

Large Language Models (LLMs) have shown significant potential in Procedural Content Generation (PCG). Yet, their application to the generation and balancing of interactive game mechanics remains largely unexplored. While PCG is a mature field, most research focuses on static assets rather than executable mechanics. Exi...

Gabriel Bauer de Oliveira, Anderson Rocha Tavares · 0 citations
#artificial intelligence Preprint Oct 2026

CriticaLLM: Using LLMs as Autonomous Game Master for Tabletop Role-Playing Games

Large Language Models (LLMs) have shown impressive capabilities across many tasks, including processing large amounts of text, following instructions, and generating ideas. This allows the use of such models in various contexts, including games and Tabletop Role-Playing Games (TTRPGs), such as Dungeons&Dragons. However...

Gaëtan Berlaimont, Justin Vanwichelen, Benoît Ronval et al. · 0 citations
Book Open access Nov 2026

From Level 1 to Victory: Progressive VLM-based Game Agents via Adaptive Multimodal Walkthroughs

Vision-language model (VLM)-based game agents have shown considerable promise. However, although these agents generally perform well in the early stages of a game, their success rates decline substantially as task difficulty increases. As a result, enabling an agent to solve advanced levels—or even complete an entire g...

Kuan Sun, Ya-Wen Zeng, Fei Liu et al. · 0 citations

CompassPlay: Rewarding the Proposer for Where It Moves the Solver

CompassPlay is introduced, a self-play method that rewards the proposer through gradient alignment that draws on a first-order approximation to learning progress and scores each eligible task without additional solver training.

Sophia Xiao Pu, Xi-Meng Sun, Jiang Liu et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.