Semantic-aligned multimodal human activity recognition using visual and audio data

Abstract

Human activity recognition (HAR) requires robust performance under heterogeneous sensing conditions for practical deployment. However, single-modality approaches are limited in capturing the rich contextual information inherent in complex human behaviors. This paper presents a semantic-aligned multimodal HAR framework that integrates visual and audio information without assuming instance-level synchronization. To address dataset heterogeneity, samples from the HMDB51 video dataset and the ESC-50 audio dataset are aligned by mapping fine-grained classes into a shared high-level activity label space. For each modality, ResNet-18-based models are trained independently using frame-based visual inputs and 64-bin Mel-spectrogram-based audio representations. During inference, the output logits of the two models are combined through score-level weighted linear fusion. Experimental results show that the proposed multimodal approach consistently outperforms unimodal baselines in terms of accuracy and Macro-F1 score, with particularly notable improvements in activity groups where environmental context plays a significant role. These findings indicate that semantic-aligned score-level fusion can improve recognition robustness even under mismatched dataset conditions.