Multimodal Emotion Recognition Using EEG and ECG: Exploring Cross-Modal Correlations and Feature Fusion Strategies

Abstract

Emotion recognition using physiological signals has attracted increasing attention in affective computing due to its objectivity and robustness, with electroencephalography (EEG) and electrocardiography (ECG) providing complementary information reflecting central and autonomic nervous system activities. This paper presents a multimodal emotion recognition framework based on EEG and ECG signals using the DREAMER dataset, where a unified processing pipeline is designed to ensure comparability between modalities through preprocessing, baseline correction, and feature extraction. Specifically, EEG features are extracted using power spectral density, while ECG features are derived from heart rate variability analysis. In addition, Pearson correlation analysis is conducted to investigate intra-modal, cross-modal, and feature-label relationships, and a support vector machine (SVM) classifier is employed for binary classification of valence, arousal, and dominance. Experimental results show that EEG and ECG exhibit low cross-modal correlation, and fused features do not consistently outperform single-modality features across tasks. These findings indicate that simple feature-level fusion may be insufficient to fully exploit multimodal information, suggesting that simple feature-level fusion may be insufficient, potentially due to feature heterogeneity.