Hybrid CNN-DNN Facial Expression Recognition in an Affective Virtual Reality (VR) and 360° E-learning Platform

Abstract

— Immersive Virtual Reality (VR) is increasingly used to support experiential learning and social presence in higher education, while 360° video offers lightweight immersive experiences accessible from standard browsers and mobile devices. However, few platforms integrate deep learning-based Facial Expression Recognition (FER) with VR and 360° e-learning in a unified architecture tailored for distance education. This paper presents an integrated e-learning platform that combines: a Unity-based multi-user VR classroom, a web-accessible 360° video learning module, and a hybrid Convolution Neural Network-Deep Neural Network (CNN-DNN) FER service that fuses appearance cues from grayscale face images with geometric descriptors from facial landmarks. We describe a proof-of-concept implementation and evaluate the FER component on public datasets (Cohn-Kanade dataset (CK+), Japanese Female Facial Expression (JAFFE), University of Oulu-Institute of Automation, Chinese Academy of Sciences (OULU-CASIA)) using subject-independent splits, and we report an exploratory pilot study with 65 university instructors who experienced a short VR lecture scenario with FER-driven avatar facial animation and provided usability and perception feedback. The fused CNN-DNN achieved 100% accuracy on CK+ (6-and 7-class settings), 88.89% on JAFFE (6 classes), and 81.25% on OULU-CASIA (6 classes), while cross-dataset performance dropped to 47.89% in the OULU→JAFFE setting, highlighting remaining generalization challenges. In the pilot study, 74% of instructors agreed that the VR scenario supported emotional communication, and 72% reported that avatar facial cues helped them notice confusion or engagement, although 93% also identified substantial integration barriers related to cost, infrastructure, and training. These results support the technical feasibility of the platform while clarifying the practical and ethical conditions required for responsible affect-aware immersive learning.