EmoSpeechBrain is introduced, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion, and shows that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.
Abstract
Multimodal emotion recognition (MER) increasingly pairs EEG with speech, treating internal neural signals and external vocal expression as informative views of affect. In practice, naive fusion underperforms the stronger single modality, because EEG artifacts inject noise that corrupts the shared representation. We introduce EmoSpeechBrain, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion. Its EEG encoder uses differential attention, taking the difference between two attention maps to cancel shared noise and isolate discriminative neural activity. An attention-based gating adapter aligns both modalities in a shared space and weights each one's contribution to the prediction. On two datasets - PME4 and EAV, EmoSpeechBrain improves MER accuracy by up to 12.9% over other state-of-the-art (SOTA) EEG encoders, and surpasses unimodal speech and EEG baselines by up to 13.1% and 23.1%. These results show that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.
An EEG-specific lightweight architecture for affective computing titled multi-scale Bidirectional Long Short-Term Memory with Attention (MS-BiLSTM-Attention) is introduced, which combines a multi-scale convolutional neural network, a Bidirectional Long Short-Term Memory, and a channel–spatial attention mechanism to joi...
Yong-Bin Zhang, Run-Yi Li, Xiu-Li Fu et al.· IEEE Access· 0 citations
Speech Emotion Recognition (SER) is a fundamental task in affective computing that enables applications such as mental health assessment, human–computer interaction, and automated customer support systems. Accurate emotion classification from speech remains challenging due to variations in speaker style, acoustic condi...
Muhammad Khalid, Abdur Rehman Tariq, Muhammad Yamin· International Bhurban Confer...· 0 citations
Cross-subject electroencephalography (EEG) emotion recognition for hearing-impaired individuals is limited by inter-subject distribution shifts, the acquisition burden of high-density systems, and overfitting on small cohorts. This study proposes RASR-Net, a Riemannian-anchored spectral-region lightweight residual netw...
Wen-Ying Wang· Journal of Computer Science...· 0 citations
This paper presents an explainable CNN–Transformer framework for subject-independent EEG emotion recognition, where the CNN component learns local temporal and spatial patterns, while the Transformer encoder uses self-attention to model relationships across the learned representation.
Venkatalakshmi M, B. M, Pavithra M et al.· International Research Journ...· 0 citations
Recognizing affective states from electroencephalography (EEG) is a core task in affective computing, with applications in mental health monitoring and public safety. Most existing approaches rely on handcrafted features or image-based transformations of EEG, while transformer models capture global context at high comp...
M. Eriş, Fatma Güneş Eriş, Orhan Yaman· Italian National Conference...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026