Skip to content
Preprint

Scaling Dexterous Manipulation via Action Representation Pretraining

Oct 2026 · 0 citations · 37 references
Computer Science

TL;DR

The proposed ARPG (Action Representation Pretraining and Generation), which explicitly separates action representation learning from action generation, achieves state-of-the-art performance on the DexJoCo benchmark and improves the average success rate on bimanual dexterous manipulation.

Abstract

Large-scale vision, language, and video pretraining has substantially improved the perception and generalization capabilities of robotic manipulation models, but action generation for complex dexterous manipulation remains constrained by the scarcity of robot demonstrations. Limited robot demonstrations must be used to learn both action representations and task-conditioned action distributions, making it difficult to adequately learn action representations and, in turn, limiting the ability of downstream generative models to generate complex dexterous actions. To address this limitation, we propose ARPG (Action Representation Pretraining and Generation), which explicitly separates action representation learning from action generation: we first pretrain an action representation encoder on large-scale human manipulation data. We then freeze the encoder and train a compact generative backbone (130M parameters) to learn task-conditioned action distributions in the encoder's representation space. Experimental results show that ARPG achieves state-of-the-art performance on the DexJoCo benchmark and improves the average success rate on bimanual dexterous manipulation from 10.1% for $\pi_{0.5}$ to 44.7%. Scaling experiments further show that broader coverage of human manipulation behaviors effectively improves downstream performance. These results indicate that action representation pretraining provides an independent scaling dimension for improving complex dexterous manipulation capabilities.

View source

Similar papers

Preprint Oct 2026

IronMan: Information-Constrained Video-Action Learning for Robot Manipulation

Video Action Models (VAMs) couple visual dynamics modeling with action generation for robot manipulation. However, video representations are not naturally suited to action generation, as exposing the action policy to excessive visual detail can impair its generalization ability. Therefore, we introduce IronMan (Informa...

Yuan-Shuo Zhang, Wen-Zhe Zhao, Zi-Xing Lei et al. · 0 citations
Conference

Robot Manipulation via Action Decomposition and Composition

Experimental results on hotdog preparation task demonstrate that the proposed action decomposition and composition framework can successfully enable both robotic arms and automated guided vehicles to perform derivative tasks, showcasing its versatility and efficiency.

Xiwen Dengxiong, Xue-Ting Wang, Yun-Bo Zhang · 0 citations
Preprint Nov 2026

CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation

Robot demonstrations contain reusable manipulation knowledge, but transferring this knowledge across embodiments requires representations that bridge differences in control interfaces. Action reconstruction alone does not ensure that similar behaviors across robots share a representation. To address this gap, we introd...

Choo Sin Wai, Da-Ke Zhong, Jing-Ye Zhang et al. · 0 citations
Sep 2026

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

WorldLine is introduced, an action-driven visual simulator that decouples transferable dynamics learning from heterogeneous action grounding and makes WorldLine a scalable and efficient visual simulator for policy evaluation and embodied planning.

Shenghe Zheng, Wen-Bo Li, Ji-Yao Zhang et al. · 1 citation
Preprint Nov 2026

MDF++: Visuo-Tactile World Action Model with End-to-end Representation Learning

Recently, world-action models (WAMs) have emerged as a powerful framework for robot manipulation. By jointly modeling action generation and future observation prediction, WAMs can act and anticipate the outcomes of the actions. However, existing WAMs typically rely on frozen pretrained encoders trained with reconstruct...

Zi-Xuan Huang, Jin-Hao Yu, Zachary Charlick et al. · 0 citations
Sep 2026

Rethinking Representations for World-Action Modeling

World-action models jointly learn robot policies and predict future observations, making the representation space an interface between control and prediction. We study the design of this space through controlled comparisons, finding that neither reconstruction fidelity nor pre-trained perceptual features alone ensure e...

Hao-Yi Jiang, Liu Liu, Xin-Jiang Wang et al. · 6 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.