Skip to content
Preprint

PiHand: Latent Hand-Object Interaction Priors for Physics-Based Policy Learning

Oct 2026 · 0 citations · 65 references
Computer Science

TL;DR

PiHand is introduced, a Physically grounded Interaction-aware Hand motion synthesis framework that trains expert policies that track mocap demonstrations in simulation, distills them into a unified hand-object interaction prior using a goal-conditioned VAE, and finally solve diverse dexterous manipulation tasks through bounded residual exploration in its latent space.

Abstract

Synthesizing physically grounded dexterous hand-object interactions is important for VR/AR, gaming, and robotics, yet remains challenging because subtle contact dynamics, diverse object geometries, and target trajectories must be jointly respected while maintaining natural grasps. Existing methods either track mocap trajectories with limited transfer to novel objects, or rely on hand-only generative priors that fail to capture coupled hand-object motion, sacrificing contact fidelity and sample efficiency. To tackle these challenges, we introduce PiHand, a Physically grounded Interaction-aware Hand motion synthesis framework. PiHand first trains expert policies that track mocap demonstrations in simulation, distills them into a unified hand-object interaction prior using a goal-conditioned VAE, and finally solve diverse dexterous manipulation tasks through bounded residual exploration in its latent space. With only a single reward and no complex curriculum training, this yields general tabletop manipulation policies that generalize to unseen objects and trajectories on GRAB and DexYCB. The same prior transfers to object catching and bimanual handover, two tasks that are rarely seen in its training data. Across all tasks, PiHand consistently outperforms baselines in all metrics. We will release the processed data and model checkpoints.

View source

Similar papers

Preprint Nov 2026

DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning

We study how to generate diverse, physically executable robot demonstrations for dexterous vision-language-action (VLA) learning. Existing demonstration-generation methods typically rely on costly teleoperated examples. Meanwhile, human hand--object datasets provide a reusable source, but existing transfer methods main...

Huaijin Pi, Ze-Hong Shen, Yong Wang et al. · 0 citations
Preprint Oct 2026

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

This work reframes physics-based retargeting as a residual learning problem, where it is demonstrated that zero-shot generalization across objects relies on structured state representations and produces a physics-based retargeter, OATH.

Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon et al. · 0 citations
Preprint Oct 2026

RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning

RLHND is proposed, a video foundation model-based hand tracking model that jointly estimates hand pose and realistic tactile information from monocular egocentric videos and achieves state-of-the-art performance across various benchmark datasets for pose estimation, while also achieving state-of-the-art performance in...

Seung-Jun Moon, Subin Jeon, Sangwoo Kim et al. · 3 citations
Preprint Oct 2026

GRASP: Global-Local Representation Alignment and Spatial-Physical Grounding for Bimanual Robotic Manipulation

Vision-Language-Action (VLA) models that directly regress joint-space actions avoid the deployment-time cost of inverse kinematics. However, their training objectives remain misaligned with task success, as joint-space losses ignore the unequal physical consequences of joint errors and visual conditioning lacks explici...

Ze-Yuan Ren, Guang-Xu Zhao, Qi Li et al. · 1 citation
#artificial intelligence Preprint Oct 2026

DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation

Achieving human-like dexterous manipulation remains a central challenge in robotics. Learning from human hand-object interaction (HOI) datasets offers a promising pathway to diverse dexterous skills, yet existing physics-based approaches train specialist policies for individual trajectories, thereby limiting skill reus...

Zhe-Xi Luo, Zhi-Xuan Xu, Rui Yang et al. · 0 citations
Preprint Oct 2026

iGPC: Generative Motion Priors for Object-Aware Humanoid Interaction

Humanoid robots operating in unstructured environments must combine robust whole-body control with the ability to perceive and physically interact with surrounding objects. While large-scale human motion data provides powerful priors for natural and versatile humanoid control, effectively transferring such priors to pe...

Anujith Muraleedharan, Abdul Ahad Butt, Nolan Fey et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.