Lightweight Gated Residual Late Fusion for Frozen Vision Priors in Vision-Only Diffusion Policies

Abstract

Visuomotor diffusion policy has shown strong capability in robotic manipulation, yet its performance depends heavily on visual representation quality. Under vision-only settings, limited expressiveness of conventional encoders bottlenecks both policy performance and convergence. Large-scale self-supervised vision foundation models such as DINOv3 offer stronger visual priors, but how to stably integrate them without increasing system complexity remains underexplored.We propose a lightweight gated residual fusion method that adaptively injects features from a frozen vision foundation model into the policy input without modifying the diffusion backbone. A channel-wise gated residual mechanism dynamically modulates pretrained priors, improving visual conditioning while preserving the original policy architecture.Experiments on four robotic manipulation tasks in ManiSkill show consistent gains under the vision-only setting: compared with CNN-based diffusion policy, the average success rate improves from 0.59 to 0.74 (+25.4%), with training convergence accelerated by up to 57.7%; compared with fine-tuning methods, the training time is 5× lower with comparable average success. Ablation studies confirm that how pretrained features are integrated matters as much as features themselves. Gate-dynamic analysis further reveals an automatic early-to-late transition from pretrained priors to task-specific representations.These results demonstrate that vision foundation model priors can be effectively leveraged through a lightweight feature-fusion module without altering the policy backbone, providing a simple yet effective path toward stronger vision-only diffusion policies.