Skip to content

Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition

Sep 2026 · 0 citations · 38 references
Computer Science Engineering

TL;DR

EmoSpeechBrain is introduced, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion, and shows that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.

Abstract

Multimodal emotion recognition (MER) increasingly pairs EEG with speech, treating internal neural signals and external vocal expression as informative views of affect. In practice, naive fusion underperforms the stronger single modality, because EEG artifacts inject noise that corrupts the shared representation. We introduce EmoSpeechBrain, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion. Its EEG encoder uses differential attention, taking the difference between two attention maps to cancel shared noise and isolate discriminative neural activity. An attention-based gating adapter aligns both modalities in a shared space and weights each one's contribution to the prediction. On two datasets - PME4 and EAV, EmoSpeechBrain improves MER accuracy by up to 12.9% over other state-of-the-art (SOTA) EEG encoders, and surpasses unimodal speech and EEG baselines by up to 13.1% and 23.1%. These results show that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.

View source

Similar papers

Open access 2026

Multi-Scale Attention-Enhanced BiLSTM Network for EEG Emotion Recognition

An EEG-specific lightweight architecture for affective computing titled multi-scale Bidirectional Long Short-Term Memory with Attention (MS-BiLSTM-Attention) is introduced, which combines a multi-scale convolutional neural network, a Bidirectional Long Short-Term Memory, and a channel–spatial attention mechanism to joi...

Yong-Bin Zhang, Run-Yi Li, Xiu-Li Fu et al. · 0 citations
Conference Open access Aug 2026

EmoNet: A Lightweight Attention-Based CNN for Speech Emotion Recognition

Speech Emotion Recognition (SER) is a fundamental task in affective computing that enables applications such as mental health assessment, human–computer interaction, and automated customer support systems. Accurate emotion classification from speech remains challenging due to variations in speaker style, acoustic condi...

Muhammad Khalid, Abdur Rehman Tariq, Muhammad Yamin · 0 citations
Open access Sep 2026

RASR-Net: Lightweight Cross-Subject EEG Emotion Recognition for Hearing-Impaired Individuals

Cross-subject electroencephalography (EEG) emotion recognition for hearing-impaired individuals is limited by inter-subject distribution shifts, the acquisition burden of high-density systems, and overfitting on small cohorts. This study proposes RASR-Net, a Riemannian-anchored spectral-region lightweight residual netw...

Wen-Ying Wang · 0 citations
Open access Sep 2026

An Explainable CNN–Transformer Framework for Subject-Independent EEG-Based Emotion Recognition

This paper presents an explainable CNN–Transformer framework for subject-independent EEG emotion recognition, where the CNN component learns local temporal and spatial patterns, while the Transformer encoder uses self-attention to model relationships across the learned representation.

Venkatalakshmi M, B. M, Pavithra M et al. · 0 citations
Open access Nov 2026

A Lightweight Hybrid-Attention Conformer for Affective State Recognition from Wearable EEG.

Recognizing affective states from electroencephalography (EEG) is a core task in affective computing, with applications in mental health monitoring and public safety. Most existing approaches rely on handcrafted features or image-based transformations of EEG, while transformer models capture global context at high comp...

M. Eriş, Fatma Güneş Eriş, Orhan Yaman · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.