MM-FusionNet: Cross-Modal Spatiotemporal Fusion for Human Action Recognition

Abstract

Multimodal human action recognition has gained momentum as real-world environments often contain occlusions, viewpoint shifts, and ambiguous motion patterns that cannot be reliably resolved from a single cue. However, effectively integrating heterogeneous signals while maintaining temporal coherence remains an open challenge. Most existing methods either operate on unimodal RGB or skeleton streams, or apply late fusion over independently learned modality scores, which limits cross-stream interaction modeling and reduces robustness when any modality is noisy or missing. To address this, we propose MM-FusionNet, a cross-modal spatiotemporal fusion framework that combines modality-specific encoders for RGB, pose, and audio with a shared Transformer-based fusion module. The fusion Transformer learns joint attention across time and modalities, enabling adaptive cue selection and noise suppression. We evaluate MM-FusionNet on HAR2 (in-house), UTD-MHAD, NTU RGB+D 60/120, and a Kinetics-400 subset, covering controlled, sensor-rich, and in-the-wild scenarios under standard cross-subject and cross-view/cross-setup protocols. MM-FusionNet consistently outperforms unimodal and late-fusion baselines, achieving 94.7% Top-1 accuracy, 94.1% F1-score, and 97.2% AUC on HAR2, and reaching 94.2% (X-Sub) and 97.1% (X-View) on NTU RGB+D 60. These results demonstrate that structured cross-modal Transformer fusion provides a scalable and robust solution for fine-grained action recognition.