Jul 2026· IEEE Robotics and Automation Letters· Vol 11, pp. 10545-10552· 0 citations· 34 references
Computer Science
TL;DR
This work proposes a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior by connecting perception and motion through shared object-level representations, and renders scenes from object-centric neural representations that integrate canonical neural fields with latent-conditioned deformations.
Abstract
Compositionality, by organizing complex behavior as combinations of simpler elements, enables robot learning that is scalable and data efficient. Leveraging this principle, we propose a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior by connecting perception and motion through shared object-level representations. We render scenes from object-centric neural representations that integrate canonical neural fields with latent-conditioned deformations, capturing positional and geometric variations in a smooth, consistent, and interpretable way. For motion generation, a temporal mixture-of-experts (MoE) employs a gating mechanism to combine object-conditioned movement primitives over time, producing complete trajectories. This spatial–temporal compositionality maintains the data efficiency of movement primitives while grounding motion in visual structure, enabling systematic generalization across diverse scene configurations. In simulation, long-horizon manipulation tasks are successfully completed using the proposed model, which requires significantly less training data than other image-based baselines. Real-world experiments further demonstrate the method's robustness to noise, its ability to generalize at the category level through language-based segmentation models, and its capacity to operate directly on 3D scene representations.
Robotic manipulation requires models that can generalize across variations in objects, scenes, and task conditions. However, collecting large-scale datasets that capture such variations in real-world robotic settings remains costly and time-consuming, making data-efficient learning an important challenge. This thesis investigates how the choice of representation can influence data-efficient generalization in robotic grasping and manipulation. First, we introduce local shape descriptors that allow grasp poses to transfer across object categories by exploiting shared geometric structure. Second, we develop neural field models that represent scenes and motions as smooth functions of latent variables learned from demonstrations. This formulation organizes demonstrations in a structured latent space, enabling motion generation from a small number of demonstrations and generalization across scene variations through interpolation. Third, we propose a potential-function-based framework for reactive motion generation, where neural fields model smooth energy functions whose gradients generate well-behaved vector fields for control. A state dependent phase formulation further enables the representation of complex motion patterns while preserving reactivity. Together, these approaches demonstrate how representation choices can improve data efficiency and generalization in robotic grasping and manipulation.
Developing generalizable robotic policies that balance inference efficiency, manipulation accuracy, and robustness remains a formidable challenge. Existing Vision-Language-Action models demand prohibitive data scales, while keyframe-based approaches struggle to reconcile the expressivity of generative models with the latency of iterative sampling. To address this trilemma, we present Flow2Act, a unified framework that integrates agglomerative perception with a deterministic one-step generative policy. Unlike prior methods relying on separate semantic encoders or iterative diffusion processes, our approach introduces three key innovations. First, we employ an agglomerative multi-teacher visual backbone that distills complementary strengths from diverse foundation models, capturing semantics, spatial structure, and segmentation to yield robust representations without task-specific pretraining. Second, we propose a conditional MeanFlow policy that parameterizes the interval-averaged velocity field. This formulation enables genuine single-step action generation, eliminating the discretization errors and computational overhead inherent in ODE-based flow matching. Third, we devise a curriculum region-aware mechanism via a Spatial-Grounded State Space Duality architecture, which progressively shifts attention from global flow stability to fine-grained contact precision. We evaluate Flow2Act on challenging simulation benchmarks and real-world robotic tasks, demonstrating significant gains in policy performance, robustness to environmental perturbations, and cross-task real-world applicability. Videos, code, and more details are available at project page.
Sen Wang, Le Wang, Hongcheng Huo et al.· IEEE Transactions on Pattern...· 0 citations
World-Action Models (WAMs) jointly learn future visual prediction and action generation, using video dynamics as a representation-learning signal for robotic manipulation. However, their video latents are primarily optimized for visual prediction and are not explicitly encouraged to preserve cross-view geometric structure or spatially localized, object-relevant semantics. We propose \textbf{GaussianWAM}, a training-time representation-enhancement framework that organizes geometric and semantic supervision through a 3D Gaussian field. Given synchronized multi-view observations, frozen geometry and vision foundation models provide depth, camera parameters, and dense semantic features. GaussianWAM binds these heterogeneous signals to shared Gaussian primitives and renders spatially aligned semantic, depth, and coverage targets, which are distilled into the current-observation representations of the WAM. All teacher models, Gaussian components, and auxiliary prediction heads are removed after training, leaving the original WAM inference path without additional modules or forward computation. On LIBERO-Plus, GaussianWAM improves FastWAM from 52.05\% to 71.29\% and Cosmos Policy from 71.52\% to 77.30\%. Direct CLIP and VGGT distillation already establishes a strong FastWAM baseline of 69.37\%, while Gaussian-field unification further improves it to 71.29\%, supporting the benefit of spatially organizing heterogeneous teacher signals. GaussianWAM also improves performance on standard LIBERO and shows positive transfer trends on RoboTwin and real-world manipulation. These results suggest that training-time Gaussian distillation provides a practical way to inject geometry- and semantics-related supervision into WAM representations without changing their deployment architecture.
Zijian Zhang, Yuqing Jiang, Weitao Zhou et al.· 0 citations
Robotic manipulation in the open world requires not only recognizing what a scene looks like, but also anticipating how its 3D structure moves under interaction. We argue that synchronized RGB, depth, and optical flow, namely RGB-DF, provide a physically grounded representation that captures the underlying 4D dynamics of a scene. Compared to 2D pixel videos, this multi-modal synergy aligns visual appearance with geometric structure and temporal motion, creating a representation space significantly closer to the low-level end-effector actions demanded by robotic systems, thereby narrowing the gap between world prediction and policy learning. Building on this insight, we introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction within one unified diffusion process. This 4D world model features a tri-branch architecture that integrates cross-modal attention with frame-wise 3D RoPE, ensuring that appearance, geometry, and motion evolve consistently. To supply training data at scale, we curate Rynn4DDataset 1.0, a massive dataset of over 254.4 million frames across egocentric human and robotic manipulation videos with high-quality pseudo-labels for depth and optical flow. We further propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the internal 4D representations of RynnWorld-4D in a single forward pass, bypassing expensive multi-step denoising, to output robot actions in a closed-loop manner. Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world dexterous bimanual manipulation tasks, particularly excelling in tasks demanding spatial precision and temporal coordination.
Haoyu Zhao, Xingyue Zhao, Siteng Huang et al.· 1 citation
Learning motion latents for robotic manipulation heavily relies on extracting motion patterns from visual sequences, yet effective action abstractions require understanding three-dimensional geometric transformations. Here, we introduce GeoMoLa (Geometry-Aware Motion Latents), which learns discrete motion latent codes by predicting how point clouds evolve during manipulation rather than reconstructing visual observations. This four-dimensional objective -- spatial geometry changing through time -- forces latent representations to encode actual physical motion rather than appearance patterns. GeoMoLa achieves state-of-the-art performance using only single-view RGB-D input, while existing methods require multi-view reconstruction, succeeding across diverse manipulation benchmarks. Our ablations reveal that geometric prediction is the key to driving performance, quantitatively validating that manipulation depends on spatial understanding. Furthermore, the learned codes exhibit effective motion abstraction: applying them to novel scenes produces physically consistent transformations regardless of visual context. Our real-world experiments also confirm this robustness capability, achieving robust manipulation with minimal demonstrations in cluttered environments where geometric reasoning determines success. Thus, we demonstrate that effective motion latents for robot control can better emerge from understanding motion through its three-dimensional effects rather than pixel-level patterns.
Yunchao Zhang, Yijia Weng, Ruizhe Liu et al.· 0 citations
Imitation learning in complex, unstructured environments remains challenging due to the difficulty of grounding perception in physically meaningful representations and the need to model multimodal action distributions. Existing approaches often rely on unstructured pixel-level feature encodings or stochastic latent-variable decoders, which can lead to brittle attention in cluttered scenes. In this work, we present a novel integration of detector-based visual representations with conditional diffusion modeling (DINO + CDP) for real-world robotic imitation learning. Our framework utilizes a DINO object detection transformer to extract spatially-grounded object-query embeddings that serve as the conditioning signal for a diffusion-based policy. A primary contribution of this work is the systematic quantification of how scene complexity—measured via image entropy—affects robotic policy performance. By comparing rigid-object baselines with complex biological plant scenes, we demonstrate that organic morphology induces a measurable increase in pixel-level uncertainty that degrades standard pixel-centric models. Our results show that DINO + CDP mitigates this degradation by grounding action generation in stable object-level features. We evaluate our approach using a fully real-world manipulation dataset collected without simulation or synthetic pre-training. To isolate the impact of our architectural choices, we conduct a comparative study within a unified framework against convolutional (CNN-MLP), transformer-patch (ViT), and latent-variable (DETR + CVAE) variants. Experimental results in a robotic-arm biocell setup demonstrate that object-query-conditioned diffusion significantly improves task success rates, produces smoother trajectories, and exhibits superior robustness to high-entropy visual inputs, establishing a scalable pathway for imitation learning in challenging agricultural domains.
Prashant Reddy Kasu, Dugan Um· Frontiers in Robotics and AI· 0 citations