AI-Powered Multimodal Human Attention Analytics with Adaptive Learning Intervention for Online Education
Abstract
The shift to online education has removed the non-verbal cues, such as gaze, posture and facial expression, that teachers use to judge whether students are engaged. Existing engagement detectors mostly rely on a single modality and send raw video to central servers, which causes accuracy problems and serious privacy risks under regulations such as the GDPR. This paper proposes an edge-computed, privacy-preserving multimodal framework that runs entirely in the learner’s browser. The system extracts 468 three-dimensional facial landmarks with MediaPipe, derives the Eye Aspect Ratio (EAR) and head-pose Euler angles with OpenCV’s solvePnP, and fuses these cues with gaze estimates and learning-platform interaction logs using a Cross-Attention Transformer. A Proximal Policy Optimization (PPO) agent then selects real-time pedagogical interventions based on estimated cognitive load. Only anonymous numeric feature vectors are processed, and no video leaves the device. In a simulated evaluation built on the DAiSEE and MPIIGaze datasets, the proposed model reaches 71.8% accuracy and a macro F1-score of 0.69 for four-level engagement recognition, outperforming CNN and LSTM baselines. Inference completes in about 18 ms per frame through WebAssembly, and the intervention policy reduces the simulated cognitive-overload rate by 34%. The results suggest that effective attention analytics and strict privacy protection can coexist in online learning.
