Skip to content

OccluDex: Hierarchical 3D Visuo-Tactile Representation Learning for Egocentric Dexterous Manipulation under Self-Occlusion

Sep 2026 · arXiv.org · Vol abs/2609.39017 · 0 citations · 30 references
Computer Science

TL;DR

OccluDex is presented, a hierarchical 3D visuo-tactile representation learning framework that integrates global geometric structure with local contact information for robust manipulation under dynamic self-occlusion during hand-object interaction and could enable humanoid egocentric object manipulation for seen and unseen objects even when the manipulating robotic hand occludes vision.

Abstract

Reliable dexterous manipulation requires continuous estimation of object geometry and hand-object contact throughout interaction. With egocentric sensing, however, the manipulating hand frequently occludes task-relevant object surfaces and contact regions, reducing the visual evidence available for state estimation and thereby making robust closed-loop control and generalization to unseen object geometries particularly challenging. To address this, we present OccluDex, a hierarchical 3D visuo-tactile representation learning framework that integrates global geometric structure with local contact information for robust manipulation under dynamic self-occlusion during hand-object interaction. OccluDex adopts multi-scale masked autoencoding to progressively encode partial 3D geometry and fuses tactile contact tokens with high-level geometric features through cross-modal attention. The encoder is pretrained from synchronized human visuo-tactile demonstrations and transferred as a frozen perceptual backbone for downstream reinforcement learning. We evaluate OccluDex on a faucet rotation task, requiring one full clockwise handle revolution, and a tabletop object reorientation task, requiring a 180-degree tabletop object reorientation without toppling. In simulation experiments, OccluDex demonstrated 12.6% higher accuracy for unseen objects and 8.3% higher accuracy for previously seen objects than the strongest state-of-the-art baseline models. Physical experiments were further performed with a Shadow Hand to demonstrate successful zero-shot sim-to-real generalization on unseen physical objects. This results could enable humanoid egocentric object manipulation for seen and unseen objects even when the manipulating robotic hand occludes vision.

View source

Similar papers

Open access Nov 2026

TransDex: Interaction Geometry Enables Robust Visuo-Tactile Dexterous Manipulation of Transparent Objects

This paper presents TransDex, an interaction-geometry-aware 3D visuo-tactile fusion motor policy for dexterous manipulation of transparent objects. Dexterous manipulation enables complex tasks but is fundamentally challenged by hand–object interaction geometric ambiguity caused by self-occlusion, severe depth noise, an...

Feng-Guan Li, Yi-Fan Ma, Chen Qian et al. · 0 citations
Preprint Sep 2026

Learning with Object-centric Representations of Tactile Interactive Perception for Robot Manipulation

Implicit object properties that are difficult to directly infer from vision, such as material, container contents, or softness, can be revealed through tactile sensing and exploratory interactions. However, because tactile signals are transient and sparse, extracting informative tactile events and effectively incorpora...

Xin-Yi Yang, Zi-Lin Si, Zhuo-Wei Xu et al. · 0 citations
Preprint Sep 2026

Praxis: Distilling Physical Interaction Priors from Egocentric Videos for Generalizable Whole-Body Manipulation

Mobile humanoid manipulation requires both reaching a usable workspace and preserving precise hand-object interactions as object poses and contact conditions change. Learning these behaviors from limited task-specific data remains challenging. To bridge this gap, we introduce Praxis, a whole-body manipulation framework...

Shu-Liang He, Rui-Yang Xu, Bo Yue et al. · 0 citations
Preprint Sep 2026

TacBPM: A Tactile-conditioned Behavior Prior Model for Dexterous Reorientation

Dexterous in-hand manipulation requires policies that coordinate high-DoF hand joints through intermittent, contact-rich interaction. Beyond target-orientation tracking, such policies must discover finger gaits that preserve object stability while adapting to geometry, anisotropy, pose, contact, and sensing changes. We...

Jie Yin, Wan-Li Xing, Ze-Yuan Zhao et al. · 0 citations

InterMASH: A Unified Geometric Representation for Grasp Synthesis

A conditional Diffusion Transformer that operates directly in the proposed InterMASH representation space and jointly generates hand geometry and contact, improving consistency and physical plausibility and shows that cross-embodiment fine-tuning with human grasp data can improve robotic grasp success and diversity.

Xuan-Ze Yang, Yu-Meng Liu, Hai-Yang Xin et al. · 0 citations
Sep 2026

Imagine then Verify: Affordance-Targeted Active Perception for Task-Oriented Grasping in Cluttered Scenes

Task-oriented grasping (TOG) requires robots to grasp functional parts of objects (e.g., the handle of a mug for pouring), yet these affordance regions are frequently occluded in cluttered scenes. Active perception via next-best-view (NBV) planning can resolve such occlusions by moving the camera for more informative o...

Jing-Zhi Cui, Xue-Feng Liu, Feng Han et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.