Performance Of Large Language Models As Hearthstone Agents
TL;DR
An LLM-driven Hearthstone agent is developed using the Sabberstone framework to evaluate several models, including GPT-4o, GPT-4o-mini, o3-mini, and GPT-5-mini, across multiple decks and prompting strategies, and results indicate that all evaluated LLMs outperform the random baseline, and GPT-5-mini achieves win rates close to the strongest numerical agents under the evaluation setting.
Abstract
This paper investigates the performance of Large Language Models (LLMs) as autonomous agents in the Hearthstone digital collectible card game. Although traditional numerical agents have demonstrated strong results in competitive game environments, the reasoning capabilities of LLM-based agents remain largely unexplored in this context. To address this gap, we developed an LLM-driven Hearthstone agent using the Sabberstone framework to evaluate several models, including GPT-4o, GPT-4o-mini, o3-mini, and GPT-5-mini, across multiple decks and prompting strategies. Our experiments compare their win rates against established numerical agents and analyze the impact of different prompting techniques, such as Chain-of-Thought (CoT), Reverse Chain-of-Thought (RCoT), ReAct, and Directed Acyclic Graph (DAG) prompting. The results indicate that all evaluated LLMs outperform the random baseline, and GPT-5-mini achieves win rates close to the strongest numerical agents under our evaluation setting. Among the prompting strategies, DAG achieved the highest win rate, while ReAct performed significantly worse. These findings highlight both the current limitations and the promising potential of LLMs for strategic reasoning and decision-making in complex, rule-based digital games such as Hearthstone. Leveraging their natural language understanding capabilities, LLMs have the potential to act as intelligent virtual companions, guiding and supporting players through the initial phases of gameplay.