Interacting with Two Dominant Hands in AR: A Study of Aimed Motions with Real and Spatially Displaced Virtual Hands

Abstract

A number of works have explored non-isomorphic hand mappings to increase the interaction space or to alter the speed/accuracy trade-off when using virtual hands in virtual reality. However, when these alterations are applied in an augmented reality context, where users perceive their real hands and interact with both real and virtual objects, the impact of these mappings remains unclear. In this paper, we investigate whether users can independently control both their real hand and a spatially displaced virtual hand, adapting their use based on task demands. To explore this, we conducted an experiment (N=24) in which participants had to touch virtual and real objects alternately. We controlled the virtual hand’s representation (anthropomorphic vs. spherical pointer) and the spatial offset between the real and the virtual representation (20 cm vs. 40 cm). We measured the transition time between real and virtual objects, the dynamics of the user’s hand (trajectory, velocity), and embodiment-related metrics (proprioceptive drift and subjective questionnaire responses). This is the first study to investigate whether such switching in an AR setup featuring one to multiple hand interactions. Our findings suggest that users can switch smoothly between real and virtual hands, highlighting the potential utility of multiple virtual hands techniques for mid-range interactions.