Automatic recognition method for communicative gestures of virtual interaction participants

Abstract

Currently, interpersonal communication increasingly takes place in virtual environments through teleconferencing systems. Despite the advantages of this format, limited video window size, unstable Internet connections, and reduced transmission quality hinder the perception of nonverbal signals exchanged by participants. During interpersonal interaction, communicative gestures serve as an important means of conveying emotions and engagement, while their automatic analysis can improve the accuracy of systems designed to recognize behavioral and psychological states of participants in virtual communication. This paper proposes a hierarchical method for communicative gesture recognition in virtual communication environments, enabling the joint recognition of both static and dynamic gestures. The proposed method is based on the sequential application of models for gesture type recognition followed by the classification of static and dynamic gestures. At the first stage, gesture type is determined as static, dynamic, or no gesture. At the second stage, specialized models are employed to recognize three classes of static gestures and three classes of dynamic gestures. Various feature types are investigated for video analysis, including body keypoints, their geometric derivative features, optical flow, and features extracted from a pretrained Vision Transformer (ViT) model. Fully connected, convolutionalrecurrent, and transformer-based neural networks are considered as classifiers. In addition, methods for improving classification accuracy under class imbalance and limited data conditions are investigated. Experimental studies were conducted using the multimodal ENERGI dataset containing recordings of group virtual communication annotated with communicative gestures. The best performance for static gesture recognition was achieved using ViT features and a convolutional-recurrent neural network, resulting in an average F1-score of 70.4 %. For dynamic gesture recognition, the most effective combination consisted of geometric derivative features and a fully connected neural network, achieving an average F1-score of 58.1 %. For gesture type recognition, the joint use of ViT features and optical flow yielded the highest average F1-score of 76.5 %. The overall average F1-score for recognizing seven communicative gesture classes reached 68.7 %, whereas the single-model approach achieved only 20.8 %. The results also demonstrate that applying class imbalance compensation techniques further improves classification performance. The results obtained confirm the effectiveness of the hierarchical approach compared to single-model classification due to the decomposition of the task into gesture type recognition and subsequent specialized classification. Different feature types were found to be most effective for different gesture categories: ViT features better capture static pose configurations, whereas geometric derivative features provide more informative representations of motion patterns. The proposed method can be applied in intelligent systems for behavioral analysis of participants in virtual communication, including engagement assessment, emotion recognition, and interaction effectiveness evaluation.