A Hybrid CNN and Mediapipe Face Mesh Framework for Real-Time Facial Emotion Recognition in Diverse Environments

Abstract

Due to its various applications, including Human-Computer Interaction (HCI), healthcare, assistive robots, intelligent surveillance, intelligent education systems, and virtual agents, interest in Facial Emotion Recognition (FER) has seen a significant increase in recent years. Nevertheless, real-time FER system development in real-world scenarios remains difficult because there are various factors that influence the FER system, such as facial expression, illumination, head position, and environment. In this paper, a hybrid solution is proposed which integrates deep-learning-based FER into a real-time face landmark detection system to enhance the system robustness in different scenarios. A CNN is trained to learn from the FER2013 facial expression dataset to extract representative features representing 7 emotions: happy, sad, angry, fear, surprised, disgusted, and neutral. Face detection and landmark estimation are done in real-time using the OpenCV and MediaPipe Face Mesh frameworks. A real-time interactive avatar uses a visual representation of the detected facial expression to communicate the recognized emotion, while spoken feedback offers a more intuitive way of interacting with the avatar. The proposed method fuses offline model learning with light weight face landmark detection to achieve high performance for interactive applications and enable efficient real-time processing. The developed framework is flexible, functional and can be used to develop real-time emotion recognition techniques in real-time interactive and assistive technology for intelligent human-robot interaction.