A Multi-Obstacle Instruction-Following Benchmark with an Obstacle-Aware Manipulation Framework

Abstract

Robotic manipulation in complex scenes requires robust reasoning and fine-grained part-level control. While recent language-guided policies have demonstrated strong manipulation capabilities, most existing benchmarks lack explicit obstacles and other sources of interference, and do not provide corresponding obstacle-aware solutions. To address this, we introduce MOBInstruct, a benchmark for part-level manipulation in cluttered environments with obstacles, along with a hierarchical framework, MOBA, for obstacle-aware execution. The MOBA framework consists of high-level modules for task planning and scene reasoning, as well as low-level modules for part-level action execution with position-aware noise initialisation. Within our framework, we evaluate representative baselines on the MOBIn-struct benchmark. Experiments demonstrate effective obstacle removal and robust task completion in multi-object scenes under diverse environmental conditions, including lighting and tabletop textures. Project website: https://j-x-chen.github.io/mobinstruct/