EMOTRACK: Real-Time Multimodal Emotion and Mental Health Monitor Using Deep Learning

Abstract

Emotional well-being is an important aspect of overall physical and psychological health; however, effectively measuring emotional states continuously in real time remains difficult because traditional methods of measurement are subjective and delay the total measurement of emotion. This paper seeks to develop and propose a framework for real-time multimodal analysis of emotion and mental health using deep learning methods. The proposed emotion monitoring system employs a combination of facial expression recognition and speech emotion recognition that enables continuous and noninvasive measurement. To extract spatial information from facial images, a Convolutional Neural Network (CNN) using ResNet50 is utilized as the feature extraction method, while temporal features from speech signals are captured through a Long Short Term Memory (LSTM) neural network. A fusion strategy performed at the decision level is applied to combine both sources of information to increase classification accuracy and resilience to variability. A web-based application implements the software to allow users to know their emotional state in real time, visualize their emotions, and generate reports. Test data shows that the facial model achieves approximately 80% validation accuracy, and the speech model achieves between 88% and 92% accuracy. The ability to provide multimodal fusion also contributes to improved prediction stability and resilience under different environmental factors. This proposed framework will provide an efficient and scalable approach to measuring real-time emotions for mental health, healthcare, and human computer interaction applications.