A unified visuo-tactile pipeline architecture for dexterous in-hand manipulation

Abstract

Fine-grained manipulation by-hand is still one of the major challenges for robotics, with recent progress-whether in visuo-tactile sensing or in representation learning or in sim-to-real transfer-being mostly parallel in time and in unrelated hardware/learning setups. There is no single architectural standard that would help compare methods re-use components, or isolate design difficulties. In this paper we suggest the Visuo-Tactile Dexterity Pipeline(VTDP), a modular organization of the dexterous manipulation stack with five linked stages: (1)multi-modal perception, (2)tactile information coding, (3)policy learning, (4)sim-to-real transition, and (5)actual task execution. With regard to each stage, the design space is analyzed, and principal trade-offs are noted, we propose specific recommendations that draw on current empirical findings. The architecture is validated by means of an analysis of 15 published systems-mapped onto the VTDP framework-which uncovers commonalities and under-studied designs. VTDP proves to be helpful both in interpreting prior work and in suggesting new methods for visuo-tactile control, i.e., for manipulation based on vision and touch.