This work proposes APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning that outperforms general-purpose VLMs and prior planning frameworks, achieving the largest gains in spatially constrained settings.
Abstract
Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility. To successfully execute a task, a robot must decompose goals, select task-relevant objects, and sequence actions, while ensuring that plans satisfy spatial constraints such as limited free space and object collisions. In this work, we propose APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning. APIVOT learns to leverage language for semantic reasoning, while using visual thoughts as imagined future states for internal verification of geometric feasibility. On long-horizon kitchen tasks, APIVOT outperforms general-purpose VLMs and prior planning frameworks, achieving the largest gains in spatially constrained settings. We find that APIVOT learns meaningful modality selection behavior, demonstrating that adaptive interleaving of vision-language thoughts improves both planning success and reasoning efficiency.
Evidence Acquisition and Feasibility Gating (EAFG) is proposed, a framework that acquires visual evidence through VLM-generated exploratory subgoals and TAMP-based execution and applies a feasibility gate to decide whether to proceed with task planning, acquire further evidence, or halt.
Tsunehiko Tanaka, Matthew Stephenson, Alistair Macvicar et al.· 0 citations
HiMe is proposed, a Hierarchical Embodied Memory framework that decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy, and introduces a dynamic knowledge system based on cross-modal semantic schemas and active management mechanisms.
Li Ji, Siyin Wang, Pengfang Qian et al.· 1 citation
Pretrained video generative models are promising backbones for visuomotor control, but their imagined futures often drift from task intent and are not reliably action-conditional. As a result, these models can be difficult to use for planning or policy extraction. To address these limitations, we propose RoboTALES, a single-stage framework that learns task-aligned simulated futures and uses them to train robot policies. Our approach introduces two key innovations: (1) a hierarchical LLM-based planner that breaks complex tasks into a sequence of subgoals to guide the model's imagination; and (2) a VLM-based critic that evaluates these ``imagined''futures and uses reward-based feedback to keep the model's internal representations focused on the goal. By anchoring the video generator in abstract reasoning, we produce temporally consistent rollouts and more coherent actions. We evaluate RoboTALES on diverse manipulation tasks from RoboCasa and LIBERO10, and show that our method consistently outperforms existing methods, especially in long-horizon tasks. Our code and models are publicly available at https://github.com/hananshafi/RoboTALES.
Hanan Gani, Tejal Kulkarni, Madhoolika Chodavarapu et al.· 0 citations
Cortex is introduced, a bidirectionally aligned embodied agent framework with a customized planning interface that conveys executable and tractable subtask plans from high-level VLM to low-level VLA and enables zero-shot completion of unseen real-world long-horizon tasks.
Jiaqi Peng, Xiqian Yu, Delin Feng et al.· 1 citation
Large Language Models (LLMs) have demonstrated strong generalization and reasoning capabilities across a wide range of domains, including embodied decision making and robotics. Despite this progress, existing reinforcement-based embodied agents often struggle with novel tasks requiring complex spatial reasoning and multi-object manipulation when relying solely on the robot’s egocentric view. In this paper, we propose SPaRL (Spatially-aware Reinforcement Learning with Language), a novel framework that explicitly integrates a structured 3D scene graph into an LLM-based reinforcement learning policy. By representing objects and their spatial relationships in a compact, semantically meaningful graph and conditioning it on task instructions, SPaRL provides the policy with explicit relational context beyond raw visual inputs. The scene graph is dynamically pruned to retain instruction-relevant objects and relations, serialized into natural language, and jointly processed with visual observations and task descriptions by a frozen LLM backbone. We evaluate SPaRL on language-conditioned rearrangement tasks in Habitat. Results show that incorporating an instruction-conditioned scene graph consistently improves performance over a vision-only LLM policy. Across curriculum training, SPaRL achieved improved performance on the Language Rearrangement benchmark. These results suggest that explicit spatial representations provide useful inductive bias for object reasoning, particularly as task difficulty increases.
Jeyoung Lee, Jaewon Lee, J. Oh et al.· 2026 23rd International Conf...· 0 citations