An Explainable Attention-based Multimodal Emotion Recognition Framework using Cross-Modal Transformer Fusion

Abstract

Emotion recognition is also important in human-computer interaction, intelligent healthcare, affective computing and smart surveillance systems. While multimodal emotion recognition has made great strides using facial expression, speech and textual cues, most current methods are based on traditional late fusion techniques which do not fully utilize the cross-modal correlation between these modalities and do not offer much interpretability. To overcome these issues, an Explainable Attention-Based Multimodal Emotion Recognition Framework Using Cross-Modal Transformer Fusion (XATF-MER) is proposed in this paper. MobileNetV2 is used to extract facial features, Wav2Vec2 for speech representation learning and BERT for contextual text embedding. Discriminative emotional attention mechanisms are added for each modality to highlight the modality-specific emotional features, and a Cross-Modal Transformer Fusion module captures complex audio-visual-textual interactions. Moreover, an adaptive confidence-aware fusion mechanism dynamically assigns the importance of each modality based on its reliability and enhances robustness when the input information is noisy and incomplete. SHAP based explainability is embedded to give transparent explanations for the predicted emotions, and to show which modality is contributing more in inference. The proposed system was benchmarked against two multimodal emotion datasets and was compared with the traditional multimodal systems using late fusion and deep learning methods. Experimental results show that the proposed method achieves an overall classification accuracy of 96.8%, a precision of 96.4%, a recall of 96.2%, and an F1-score of 96.3%, that are better than the existing baseline models. Beyond this, the adaptive fusion strategy enhances the recognition robustness in the missing or noisy modalities and the explainability module offers meaningful visual interpretation without compromising the real-time performance. The results presented here show that the proposed framework is an effective, reliable and interpretable solution for more advanced emotion recognition systems, which can be used in healthcare, intelligent virtual assistants, education and human-computer interaction.