MAFN: Multimodal Attention Fusion Network for Emotion Recognition

Abstract

Emotion recognition from physiological signals is a rapidly advancing area within affective computing and human-computer interfacing. This study presents a novel technique for emotion recognition that leverages Electroencephalogram (EEG) and Electrocardiogram (ECG) signals. It is observed that as emotions change, the patterns of EEG and ECG signals also change. This observation inspired us to propose a new Multimodal Attention Fusion Network (MAFN). This MAFN integrates Bidirectional Long Short-Term Memory (BiLSTM) and self-attention mechanisms to extract effective features for emotion classification. In this work, the adapted BiLSTM extracts spatial and temporal features, while a self-attention network extracts contextual features to improve the classification performance. To evaluate the model's performance, three benchmark datasets, DREAMER, AMIGOS, and Multimodal, are used to validate the proposed and existing models with a 5-fold nested cross-validation approach. Extensive experiments and analyses across all three datasets confirm the effectiveness of this approach in emotion classification. A comparative study of the proposed model with the state-of-the-art emotion recognition models shows that our work consistently surpasses state-of-the-art models on different benchmark datasets in terms of classification rate.