Predicting 3D consistent waypoints (3DWay) from multi-view images as generating multi-view consistent 2D waypoints followed by geometric triangulation enables explicit 3D motion specification while preserving the strong priors of pretrained VLMs.
Abstract
Intermediate representations are key to bridging the modality gap between generalizable manipulation policies and large-scale pretrained vision-language models (VLMs). Among these, trajectory-based representations compactly represent motion-relevant cues, yet most existing approaches predict trajectories in 2D image space, resulting in intrinsic 3D ambiguity. Moreover, using 2D trajectories with depth still leaves the free-space waypoints ambiguous, limiting reliable 3D reasoning. To address this, we propose predicting 3D consistent waypoints (3DWay) from multi-view images. By reformulating 3D waypoints prediction as generating multi-view consistent 2D waypoints followed by geometric triangulation, we enable explicit 3D motion specification while preserving the strong priors of pretrained VLMs. The predicted waypoints can guide existing VLA models for better generalization or be directly executed on simple tasks. Extensive experiments show that 3DWay substantially improves 3D spatial grounding and vision-language reasoning, demonstrating strong potential for generalizable robot manipulation. Codes will be released at https://github.com/ziqin-h/3DWay.
This work introduces a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics, and trains DepthWorld, a Stable Video Diffusion-based world model that jointly...
Jai Bardhan, Josef Sivic, Vladimír Petrík· 2 citations
FlowVLA is introduced, a reconstruction-guided, pluggable 3D flow learning framework aimed at empowering VLA models with an inherent understanding of action-relevant dynamics, designed to utilize temporally aligned 3D flow only during training.
Junhyeong Go, Ming-Yu Jung, Woo-Bin Im et al.· 0 citations
Spatial Grafting is proposed, a versatile, lightweight spatial module that binds frozen reconstruction features to metric, robot-relative geometry and injects them into the flow-matching action expert through cross-attention without modifying the host's perceptual pathway, so the host retains the full benefit of its pr...
RoGSW4RLD is introduced, a feed-forward framework that lifts synchronized multi-camera rollouts into a unified, time-queryable metric 4D Gaussian field and directly reconstructs the visual future generated by existing world models.
Jin Hyun Kim, Min Young Kim, Soohwan Song et al.· arXiv.org· 0 citations
Enabling navigation agents to follow human instructions and reach target locations in complex environments is a fundamental requirement for embodied intelligence. Despite recent advances in Vision-and-Language Navigation (VLN) that leverage Vision-Language Models (VLMs) for environmental and semantic understanding, two...
Yi-Hao Wu, Chen-Huan Cai, Li-Qi Yan et al.· Proceedings of the 34th ACM...· 0 citations
Geometric structure is essential for robotic manipulation and understanding the 3D world. Most VLA models rely on 2D representations of current observations, while geometry-aware approaches often require additional 3D information. However, geometry inferred independently from individual observations is often discarded...
Qin-Fang Zhang, Yi-Fan Luo, Hao-Yuan Shi et al.· 2 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduJul 30, 2026
From feet to fingertips — we are teaching robots intelligent whole-body control, fine dexterity, and teamwork to complete a broad range of complex tasks.