Spatial-Frequency Hypergraph Neural Network for EEG-fNIRS Emotion Recognition
TL;DR
An EEG-fNIRS emotion recognition framework based on a Spatial-Frequency Hypergraph Neural Network (SF-HGNN) that achieves high-order interaction between the complementary information of the two modalities and was validated on the public ENTER dataset.
Abstract
Background/Objectives: Hybrid EEG-fNIRS emotion recognition aims to accurately identify an individual’s emotional state by analyzing neurophysiological signals and constitutes an important research direction in affective brain–computer interfaces and human–computer interaction. In recent years, EEG-fNIRS emotion recognition has advanced from handcrafted feature extraction and shallow fusion to deep learning and graph-based modeling. However, most existing methods rely on predefined fixed frequency-band partitioning and second-order graph structures that only support pairwise connections, making it difficult to accommodate inter-subject frequency variability and to characterize high-order brain network relationships such as multi-channel synergistic activation within a frequency band and cross-frequency coupling. Methods: To address these issues, this paper proposes an EEG-fNIRS emotion recognition framework based on a Spatial-Frequency Hypergraph Neural Network (SF-HGNN). First, a Dynamic Frequency Band Decomposition module is designed to achieve adaptive optimization of the EEG and fNIRS frequency bands; second, a Multi-scale Temporal Convolution module extracts temporal features at different time scales; third, a Spatial-Frequency Adaptive Hypergraph Convolution module is constructed to model intra-band cross-channel spatial synergy and channel-wise cross-frequency coupling; and finally, a Cross-Modal Attention Fusion mechanism achieves high-order interaction between the complementary information of the two modalities. The proposed method was validated on the public ENTER dataset comprising 50 participants and four emotion categories (sadness, happiness, fear, and calm). Results: Experimental results show that SF-HGNN achieves accuracies of 82.94% and 68.85% in subject-dependent and subject-independent experiments, respectively; ablation studies and visualization analyses further verify the effectiveness of each module and the interpretability of the model. Conclusions: Future work will focus on validation with larger-scale data and improving cross-subject domain generalization.