The results suggest that SPEEDRUNBENCH is a useful testbed for studying agents's strategy formation capabilities as well as being a saturation-resistant evaluation measure, as there is almost always a faster completion time waiting to be discovered.
Abstract
Frontier LLM agents have been shown to be capable of solving increasingly complex tasks for which humans have measurable solutions. This begs the pertinent question of whether LLM agents can go beyond what humans have already solved. The ability to develop sophisticated strategies to tackle consequential problems becomes paramount as well-trodden, human-developed solutions become insufficient for problems for which we lack context or enough training data. We study agents'capability of such strategy formation through the communal practice of video game speedrunning. In speedrunning, practitioners compete to find the fastest way to complete a video game under certain conditions, and in so doing uncovering interesting unorthodox play styles that require a thorough understanding and mastery of the underlying game mechanics. We introduce SPEEDRUNBENCH, a benchmark that evaluates frontier LLM agents across 9 different games. To perform well in this benchmark, agents must repeatedly improve their strategy, reflect on their performance, exploit their gained knowledge, and reason across a long-horizon of actions to improve on an increasingly difficult problem: being faster than themselves and everyone else. Our experiments show that while frontier agents approach human world records in simple platformer games, they remain behind human performance on longer, more complex games under practical budgets. These results suggest that SPEEDRUNBENCH is a useful testbed for studying agents'strategy formation capabilities as well as being a saturation-resistant evaluation measure, as there is almost always a faster completion time waiting to be discovered.
GameBoyWorlds is introduced, a testbed for agentic self-improvement in video games and it is shown that contemporary approaches to self-improvement are lacking, with world modelling and autonomous skill discovery failing, and a novel strategy that uses curiosity-based exploration to write guides achieving only partial...
Dhananjay Ashok, Adam Shen, Aslan H. Feng et al.· arXiv.org· 1 citation
This work introduces LongPuzzleBench, 114 levels in six puzzle games played through native GUI actions, where one objective can take a human over a thousand actions on persistent boards and dead ends go unannounced.
Bing-Xu Zhang, Hao-Chuan Lu, Zong-Jie Li et al.· arXiv.org· 2 citations
Large Language Models (LLMs) have shown significant potential in Procedural Content Generation (PCG). Yet, their application to the generation and balancing of interactive game mechanics remains largely unexplored. While PCG is a mature field, most research focuses on static assets rather than executable mechanics. Exi...
Gabriel Bauer de Oliveira, Anderson Rocha Tavares· Proceedings of the AAAI Conf...· 0 citations
Large Language Models (LLMs) have shown impressive capabilities across many tasks, including processing large amounts of text, following instructions, and generating ideas. This allows the use of such models in various contexts, including games and Tabletop Role-Playing Games (TTRPGs), such as Dungeons&Dragons. However...
Gaëtan Berlaimont, Justin Vanwichelen, Benoît Ronval et al.· 0 citations
Vision-language model (VLM)-based game agents have shown considerable promise. However, although these agents generally perform well in the early stages of a game, their success rates decline substantially as task difficulty increases. As a result, enabling an agent to solve advanced levels—or even complete an entire g...
Kuan Sun, Ya-Wen Zeng, Fei Liu et al.· Proceedings of the 34th ACM...· 0 citations
CompassPlay is introduced, a self-play method that rewards the proposer through gradient alignment that draws on a first-order approximation to learning progress and scores each eligible task without additional solver training.
Sophia Xiao Pu, Xi-Meng Sun, Jiang Liu et al.· arXiv.org· 0 citations