From Level 1 to Victory: Progressive VLM-based Game Agents via Adaptive Multimodal Walkthroughs
Abstract
Vision-language model (VLM)-based game agents have shown considerable promise. However, although these agents generally perform well in the early stages of a game, their success rates decline substantially as task difficulty increases. As a result, enabling an agent to solve advanced levels—or even complete an entire game—remains a formidable challenge. In fact, human players often consult external walkthroughs to overcome difficult levels. For VLM-based agents, however, the multimodal, unstructured, and often redundant nature of such walkthroughs presents significant obstacles. In this paper, we propose Walkthrough Agent, a novel framework that enables agents to adaptively identify relevant clues from redundant walkthroughs and, through reasoning, derive appropriate actions for successful execution, thereby completing more challenging game levels. Specifically, the framework consists of three modules: 1) Gather, which automatically collects raw multimodal walkthroughs; 2) Reasoner, which employs a three-stage strategy—locating, reasoning, and action—to progressively identify relevant clues and infer appropriate actions; and 3) Executor, which directly applies the generated action commands to interact with the game environment. Extensive experiments across three game environments demonstrate that our framework substantially improves success rates on challenging levels. Moreover, using only 1,000 verified samples to fine-tune Qwen3-VL-8B, our method substantially outperforms the same-backbone RAGEN baseline and achieves competitive performance relative to closed-source models.