3D Gaussian Digital Twins for View-Robust 3D Visuomotor Policy Learning on Articulated Objects

Abstract

Single-view 3D visuomotor policies based on point-cloud observations, such as 3D Diffusion Policy, often suffer from limited viewpoint generalization due to partial observability and occlusion. This is particularly challenging for articulated objects whose internal motion induces geometry changes. To address this problem, we propose an articulation-driven digital twin augmentation approach that couples reconstruction, online articulation-state estimation, and observation generation within a shared 3D Gaussian Splatting (3DGS) model. Offline, we reconstruct an articulation-aware 3DGS part model and parameterize the articulation. Online, we optimize joint states through differentiable rendering conditioned on the base pose, then transform pre-sampled part point clouds using the estimated configuration. Fusing the object point cloud with robot geometry enables real-time augmentation from partial to global observations using single-camera input during execution. Experiments on the complete pipeline in simulation and the real world demonstrate accurate articulation tracking and improved viewpoint generalization of 3D Diffusion Policy.