SAIN: A Hierarchical Dynamics-Aware EEG–fNIRS Framework for Continuous Affect Estimation

Abstract

Continuous affect estimation from electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) aims to recover time-varying Valence and Arousal from heterogeneous physiological signals. Their differences in channel organisation, response latency, temporal statistics, and noise characteristics make homogeneous fusion and single-timescale modelling insufficient. We propose the Structured Affective Interaction Network (SAIN), a hierarchical framework comprising Structured-Axial Physiological Encoder (SAPE), Adaptive Cross-Modal Interaction (ACI), and Dual-Rate Evolution (DRE). SAPE independently preserves modality-specific channel relations while integrating time-indexed and trial-level channel-feature context. ACI explicitly models feature-wise selection, modality discrepancy, and coordinated responses to produce a fused physiological representation and a direct physiological estimate. DRE then uses a training-derived Group Affective Trajectory (GAT) as coarse temporal context and combines dimension-wise adaptive gain, fast correction, and a retention-controlled slow recurrent state. Experiments demonstrate that SAIN outperforms reproduced baselines and independently trained unimodal variants. Modality controls support the complementary integration of EEG and fNIRS, channel-masking analyses reveal spatially structured physiological dependence, and temporal ablations establish the distinct contributions of fast correction and slow recurrent accumulation. These results jointly support the structured encoding, adaptive interaction, and dual-rate evolution underlying SAIN.