Skip to content
Open access

Progressive Evolution of Emotion Detection: From Unimodal Baselines to a Quad-Modal Dynamic Fusion Architecture

Aug 2026 · Cureus Journal of Computer Science · Vol 3 · 0 citations · 22 references

TL;DR

The architecture, while forgoing the precision of a studio-lit, well-audio-recorded environment, is able to sacrifice some accuracy for the robustness required to function in a real-world human-computer interaction environment.

Abstract

The existing emotion detection systems are either audio or video centric. While such approaches have proven effective in controlled environments like a lab or a studio, these systems fail to perform under real-world conditions such as low light, audio interference, mispronunciation, and other environmental factors. The objective of our research is to design a quad-modal emotion detection system that combines four sub-models (static facial images, dynamic facial expressions, speech tone, and natural language processing [NLP] of the spoken words) to design a robust model that can perform under real-world conditions. We propose a dynamic weighting algorithm that biases the model towards a certain modality if the conditions are favorable (e.g. switch to audio-only if the video is obstructed) using cross-modal multi-head attention. We optimise the system to run on a CPU (due to the lack of a CUDA-enabled NVIDIA GPU) by using an asynchronous state buffer for the heavy NLP pipeline to maintain a high enough video frame rate. Our model achieved an accuracy of 61.21% (95% confidence interval [CI]: 58.77%-63.65%) with a mean area under the curve of 0.90 (95% CI: 0.88-0.92) across six emotion classes on real-world data (RAVDESS, SAMM, CREMA-D) during training and testing while maintaining performance under visual and audio deprivation conditions. Our architecture, while forgoing the precision of a studio-lit, well-audio-recorded environment, is able to sacrifice some accuracy for the robustness required to function in a real-world human-computer interaction environment.

Read PDF

Similar papers

Book Open access Oct 2026

Light-ED: Lightweight Multimodal Emotion Detection using Enhanced EfficientNet

A unified and resource-efficient framework that integrates multimodal fusion and emotion classification within a single architecture, and introduces a lean cross-modal cross-attention (CMCA) mechanism to enable effective interaction between audio and visual modalities with low computational cost.

Wamika Jha, Mea Wang, U. Alim et al. · 0 citations
Sep 2026

Emotionnet: A Deep Learning Framework for Real-Time Video-Based Facial Emotion Recognition

EmotionNet is a neural network that is used to recognize facial emotions in real-time and under dynamic and unconstrained conditions using videos. The suggested system combines both spatial features extraction and time modeling to ensure the capture of facial appearance and motion-based emotion features. A convoluted n...

Amit Rehapade · 0 citations
Open access Aug 2026

Lightweight and robust audio-visual emotion recognition via multi-scale mamba temporal modeling and quality-aware expert fusion

Experiments demonstrate that the proposed lightweight audio-visual emotion recognition framework achieves competitive recognition accuracy with significantly fewer parameters and lower computational cost.

Tianxing Zhang, Hadi Affendy Bin Dahlan, Fadhilah Rosdi et al. · 0 citations
Open access Aug 2026

FER20E: An Extended Facial Expression Recognition Dataset With 20 Discrete Emotions

The FER20E dataset provides a comprehensive benchmark for advancing emotion recognition in unconstrained and real-world scenarios, and a data annotation tool (DL-DAT) that follows a semi-automated, human-in-the-loop pipeline to enable scalable and reliable annotation.

Kuldeep Singh Yadav, Lalan Kumar · 2 citations
Open access Sep 2026

ARECA-Lite: A lightweight modified ArecaNet with reduced complexity for real-time robust facial emotion recognition

Facial emotion recognition (FER) in e-learning environments faces a persistent tension between recognition performance and deployment efficiency; high-performing architectures typically impose prohibitive computational costs that preclude real-time use on commodity hardware. This paper presents ARECA-Lite, a lightweigh...

Mustapha Abdelkader Laoumir, Amina Kinane Daouadji, Fatima Bendella · 0 citations
Review Open access Aug 2026

A Systematic Review of Emotion Recognition: From Unimodal Signals to Multimodal Integration

These findings reveal that multimodal systems, which fuse visual, acoustic, linguistic, linguistic, and physiological signals, consistently outperform unimodal counterparts, achieving accuracy levels above 85% on benchmark datasets.

B. Bashir, Zayyanu Yunusa · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.