SPaRL: Spatially-aware Reinforcement Learning with Language

Abstract

Large Language Models (LLMs) have demonstrated strong generalization and reasoning capabilities across a wide range of domains, including embodied decision making and robotics. Despite this progress, existing reinforcement-based embodied agents often struggle with novel tasks requiring complex spatial reasoning and multi-object manipulation when relying solely on the robot’s egocentric view. In this paper, we propose SPaRL (Spatially-aware Reinforcement Learning with Language), a novel framework that explicitly integrates a structured 3D scene graph into an LLM-based reinforcement learning policy. By representing objects and their spatial relationships in a compact, semantically meaningful graph and conditioning it on task instructions, SPaRL provides the policy with explicit relational context beyond raw visual inputs. The scene graph is dynamically pruned to retain instruction-relevant objects and relations, serialized into natural language, and jointly processed with visual observations and task descriptions by a frozen LLM backbone. We evaluate SPaRL on language-conditioned rearrangement tasks in Habitat. Results show that incorporating an instruction-conditioned scene graph consistently improves performance over a vision-only LLM policy. Across curriculum training, SPaRL achieved improved performance on the Language Rearrangement benchmark. These results suggest that explicit spatial representations provide useful inductive bias for object reasoning, particularly as task difficulty increases.