Real-Time Interest Level Estimation Using YOLO-Based Face Detection and CNN-Based Emotion Recognition in an Interactive Training System

Abstract

This study presents the development and application of a real-time interest level estimation system for interactive training settings. The system incorporates a Convolutional Neural Network (CNN)-based emotion recognition model and You Only Look Once version 12 (YOLOv12)-based face detection for efficient and accurate facial detection, whereas RetinaFace is used for enhanced face extraction during preprocessing. The CNN is trained on a locally collected Indonesian facial-emotion dataset with 31 participants, each contributing two videos, resulting in a total of 358 videos, a raw dataset of 529,834 files, and a cleaned dataset of 91,219 files. This dataset was used to improve the model's robustness and generalizability across various facial features and cultural expressions. YOLOv12-based face detection is used to extract faces from videos during the data preprocessing stage. The CNN model is then used to classify the extracted faces into seven primary emotional states: happy, sad, angry, surprised, neutral, disgusted, and frightened. This research investigates facial expressions as important indicators of emotional state in educational settings and proposes a mapping between learning engagement measures and conventional Facial Expression Recognition (FER) categories. A rule-based logic model grounded in affective computing theory is then used to map these emotional states to corresponding interest levels. For example, "sad," "frightened," or "disgusted" indicate apathy, whereas "happy" and "surprised" are considered indicators of strong interest. The proposed model demonstrates strong performance in real-time training applications, achieving an accuracy of 98%. Trainers can evaluate learner engagement during sessions through a real-time feedback dashboard. Based on participant interest levels, this system enables instructors to adjust instructional strategies such as pacing, topic selection, or teaching methodology.