Skip to content

3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints

Sep 2026 · 0 citations · 71 references
Computer Science

TL;DR

Predicting 3D consistent waypoints (3DWay) from multi-view images as generating multi-view consistent 2D waypoints followed by geometric triangulation enables explicit 3D motion specification while preserving the strong priors of pretrained VLMs.

Abstract

Intermediate representations are key to bridging the modality gap between generalizable manipulation policies and large-scale pretrained vision-language models (VLMs). Among these, trajectory-based representations compactly represent motion-relevant cues, yet most existing approaches predict trajectories in 2D image space, resulting in intrinsic 3D ambiguity. Moreover, using 2D trajectories with depth still leaves the free-space waypoints ambiguous, limiting reliable 3D reasoning. To address this, we propose predicting 3D consistent waypoints (3DWay) from multi-view images. By reformulating 3D waypoints prediction as generating multi-view consistent 2D waypoints followed by geometric triangulation, we enable explicit 3D motion specification while preserving the strong priors of pretrained VLMs. The predicted waypoints can guide existing VLA models for better generalization or be directly executed on simple tasks. Extensive experiments show that 3DWay substantially improves 3D spatial grounding and vision-language reasoning, demonstrating strong potential for generalizable robot manipulation. Codes will be released at https://github.com/ziqin-h/3DWay.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

DepthWorld: 3D World Model for Robot Manipulation

This work introduces a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics, and trains DepthWorld, a Stable Video Diffusion-based world model that jointly...

Jai Bardhan, Josef Sivic, Vladimír Petrík · 2 citations
Preprint Oct 2026

FlowVLA: Enabling World Dynamics Understanding through Pluggable 3D Flow Learning

FlowVLA is introduced, a reconstruction-guided, pluggable 3D flow learning framework aimed at empowering VLA models with an inherent understanding of action-relevant dynamics, designed to utilize temporally aligned 3D flow only during training.

Junhyeong Go, Ming-Yu Jung, Woo-Bin Im et al. · 0 citations

Spatial Grafting: Grounding 3D Features for Flow-Matching Robot Policies

Spatial Grafting is proposed, a versatile, lightweight spatial module that binds frozen reconstruction features to metric, robot-relative geometry and injects them into the flow-matching action expert through cross-attention without modifying the host's perceptual pathway, so the host retains the full benefit of its pr...

Ding-Sheng Liu, Yang-Zheng Wu, Mahboubeh Asadi et al. · 0 citations
Sep 2026

RoGSW4RLD: Feed-Forward 4D Gaussian Lifting for Robot World Model Rollouts

RoGSW4RLD is introduced, a feed-forward framework that lifts synchronized multi-camera rollouts into a unified, time-queryable metric 4D Gaussian field and directly reconstructs the visual future generated by existing world models.

Jin Hyun Kim, Min Young Kim, Soohwan Song et al. · 0 citations
Book Open access Nov 2026

ShapeNav: 3D Geometric Perception and Reasoning for Multi-Floor Vision-and-Language Navigation

Enabling navigation agents to follow human instructions and reach target locations in complex environments is a fundamental requirement for embodied intelligence. Despite recent advances in Vision-and-Language Navigation (VLN) that leverage Vision-Language Models (VLMs) for environmental and semantic understanding, two...

Yi-Hao Wu, Chen-Huan Cai, Li-Qi Yan et al. · 0 citations
Preprint Oct 2026

GeoMem-VLA: Internalizing 3D Scene Understanding into Temporal Memory for Vision-Language-Action Models

Geometric structure is essential for robotic manipulation and understanding the 3D world. Most VLA models rely on 2D representations of current observations, while geometry-aware approaches often require additional 3D information. However, geometry inferred independently from individual observations is often discarded...

Qin-Fang Zhang, Yi-Fan Luo, Hao-Yuan Shi et al. · 2 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.