Aug 2026· Cureus Journal of Computer Science· Vol 3· 0 citations· 22 references
TL;DR
The architecture, while forgoing the precision of a studio-lit, well-audio-recorded environment, is able to sacrifice some accuracy for the robustness required to function in a real-world human-computer interaction environment.
Abstract
The existing emotion detection systems are either audio or video centric. While such approaches have proven effective in controlled environments like a lab or a studio, these systems fail to perform under real-world conditions such as low light, audio interference, mispronunciation, and other environmental factors. The objective of our research is to design a quad-modal emotion detection system that combines four sub-models (static facial images, dynamic facial expressions, speech tone, and natural language processing [NLP] of the spoken words) to design a robust model that can perform under real-world conditions. We propose a dynamic weighting algorithm that biases the model towards a certain modality if the conditions are favorable (e.g. switch to audio-only if the video is obstructed) using cross-modal multi-head attention. We optimise the system to run on a CPU (due to the lack of a CUDA-enabled NVIDIA GPU) by using an asynchronous state buffer for the heavy NLP pipeline to maintain a high enough video frame rate. Our model achieved an accuracy of 61.21% (95% confidence interval [CI]: 58.77%-63.65%) with a mean area under the curve of 0.90 (95% CI: 0.88-0.92) across six emotion classes on real-world data (RAVDESS, SAMM, CREMA-D) during training and testing while maintaining performance under visual and audio deprivation conditions. Our architecture, while forgoing the precision of a studio-lit, well-audio-recorded environment, is able to sacrifice some accuracy for the robustness required to function in a real-world human-computer interaction environment.
A unified and resource-efficient framework that integrates multimodal fusion and emotion classification within a single architecture, and introduces a lean cross-modal cross-attention (CMCA) mechanism to enable effective interaction between audio and visual modalities with low computational cost.
Wamika Jha, Mea Wang, U. Alim et al.· Proceedings of the 28th Inte...· 0 citations
EmotionNet is a neural network that is used to recognize facial emotions in real-time and under dynamic and unconstrained conditions using videos. The suggested system combines both spatial features extraction and time modeling to ensure the capture of facial appearance and motion-based emotion features. A convoluted n...
Amit Rehapade· Natural Resources for Human...· 0 citations
The FER20E dataset provides a comprehensive benchmark for advancing emotion recognition in unconstrained and real-world scenarios, and a data annotation tool (DL-DAT) that follows a semi-automated, human-in-the-loop pipeline to enable scalable and reliable annotation.
Facial emotion recognition (FER) in e-learning environments faces a persistent tension between recognition performance and deployment efficiency; high-performing architectures typically impose prohibitive computational costs that preclude real-time use on commodity hardware. This paper presents ARECA-Lite, a lightweigh...