MultiModal Emotion Detection

Abstract

Emotion recognition plays a critical role in affective computing systems that aim to understand human behavior through observable signals. While uni-modal approaches based on audio, facial expressions, or body language provide useful cues, their performance often degrades under real-world conditions due to noise, occlusions, and modality-specific limitations. This paper presents a multi-modal emotion detection framework that integrates audio, facial, and body-language information using a learnable weights strategy. Each modality is processed by an independently trained deep learning model tailored to its feature characteristics, and predictions are fused at the score level with robustness to missing or unreliable inputs. Emotions are inferred over successive non-overlapping 3-second temporal windows, enabling time-resolved analysis of emotional dynamics. Experiments conducted on a custom multimodal dataset demonstrate that the proposed fusion model achieves an overall accuracy of 77% and provides stable per-class performance across diverse emotions. In addition to static classification, the system produces continuous emotion probability spectra and predicted emotion timelines, offering interpretable insights into emotional transitions over time. The proposed approach is particularly suited for therapeutic and behavioral assessment scenarios where robustness and temporal interpretability are essential.