MICA-Net: A Multimodal Cross-Attention Network for Human Action Recognition

Abstract

Automatic human action recognition (HAR) has become the most active research topic in recent years due to its broad applications, ranging from health monitoring and video surveillance to human-robot interaction. In this paper, we introduce a novel action recognition method, named MICA-Net, which combines data from multiple sensors to improve the efficiency of the HAR model. MICA-Net is composed of a lightweight 3D CNN, optimized for mobile devices, to extract visual features and a co-attention network that integrates a 2D CNN with a transformer to extract motion features. These extracted features are then continuously fused through dynamic gated Joint Cross-Attention Modules (JCAMs). These modules capture the intra- and inter- modal relationship while adaptively learning the contribution of each modality across different scenarios. We evaluated our recognition model on four publicly available multimodal datasets, MMAct, UESTC-MMEA-CL, UTD-MHAD, and MuWiGes. On MMAct, our model achieves an impressive F1-score of 89.24% with Cross-Subject and 96.57% with Cross-Session, outperforming current state-of-the-art methods. Similarly, on the UESTC-MMEA-CL, UTD-MHAD, and MuWiGes datasets, it achieves outstanding accuracy of 99.24%, 94.72%, and 98.98%, respectively. To demonstrate the practicality of the model, we design a new compact version of a wrist-worn sensor device with Wi-Fi connectivity to an edge device, enhancing usability in human-machine interaction applications. Real-time deployment indicates its potential for real implementation in the future. Our code is publicly available at https://anonymous.4open.science/r/MICA-Net-682E.