Causal Abstraction Learning for Multi-Modal Grounded Planning

Abstract

Recent advances in multimodal embodied agents have enabled long-horizon planning in visually rich environments via natural language. Yet, their generalization remains brittle when task instructions deviate from familiar examples, exposing a reliance on surface imitation rather than structural understanding. We propose Causal Abstraction Learning for Multi-Modal Grounded Planning (CALM), a framework that enhances planning agents with the ability to discover and exploit causal regularities across tasks. CALM incrementally develops a causal library by abstracting precondition–effect structure from successful executions, yielding compact representations that emphasize stable dependencies beyond incidental context. When execution diverges from expectation, these abstractions are refined through contrastive causal reasoning, enabling targeted adjustments that resolve underlying mechanism mismatch. The resulting structure serves as a transferable prior for planning in novel settings, integrating perceptual cues with mechanism-informed knowledge. Without retraining or task-specific heuristics, CALM generalizes robustly and efficiently to linguistic and perceptual variation. Experiments on ALFRED and VirtualHome demonstrate consistent gains, highlighting causal abstraction as a scalable inductive bias for grounded planning.