Using Hand Gestures for Multimodal Interactions: Foundations, Methods, and Emerging Challenges

Abstract

Hand gestures are a natural and expressive modality for human communication and are increasingly used in multimodal interaction, including human-computer interaction, extended reality, robotics, assistive technologies, and embodied AI. However, designing robust gesture-based interfaces remains challenging due to issues related to gesture acquisition, tracking quality, continuous recognition, semantic interpretation, ergonomics, and real-time deployment. In this tutorial, we will present methods and challenges involved in developing effective gestural interaction systems. We will first introduce the role of hand gestures in multimodal interaction and discuss how gestures can complement other modalities such as speech, gaze, body motion, and object interaction. We will then present current technologies for hand and finger tracking, including vision-based, sensor-based, and wearable approaches, with a focus on their limitations in terms of accuracy, robustness, latency, and data quality. A central part of the tutorial will focus on hand gesture recognition, covering segmented and continuous recognition, gesture representations, learning-based methods, benchmarks, and tools for real-time recognition. We will also introduce recent advances in hand motion generation, with emphasis on multimodal conditioning from text, audio, body motion, and objects, as well as key challenges related to realism, physical plausibility, and real-time performance. Overall, the tutorial will provide participants with a concise but comprehensive view of how hand gestures can be captured, recognized, interpreted, and generated in multimodal interactive systems.