Domain-Adaptive Emotion Dual-Space Network for Reproducible Multimodal Teacher Emotion Recognition

Abstract

In smart education, it is very important to understand teachers' emotions during teaching, as the emotions of teachers influence students' participation, learning ability and classroom atmosphere. Today there are 3 problems that hinder the methods used. First, the direction of the head is not considered to have any real emotional information and is not used. Second, the available models learn from emotions expressed by actors, but not from the way teachers truly behave in the real classroom. Thirdly, the best TER system is tested with a private data set that no-one else has access to. We address all three with the Domain-Adaptive Emotion Dual-Space Network (DA-EDSN) which is a modification of EDSN [1] in four aspects. Unlike CMD, which makes a general comparison between the two modalities, Cross-Modal Attention identifies specific dimensions of one modality related to another. Instead, Gradient Reversal Layer (CD) is used to remove the Gradient Reversal and to push similar utterances from the same class to each other and push dissimilar utterances from different classes apart in the fused space. Hierarchical Attention Fusion takes two streams: one from audio and text, and another from MOCAP and MM-TBA, and fuses them with audio weights which are selected per sample. No labelled target data is required, a Domain Adaptation Module maps the source features toward the real domain behavioral distribution via minimizing the Maximum Mean Discrepancy (MMD). Last but not least, head motion recorded using a MOCAP system is a fourth, independent modality (a first for any published TER system). On public IEMOCAP, DA-EDSN achieves an absolute improvement of 2.4% to 3.1%, to ACC 0.816 and WF1 0.815 (a 27.2% improvement over the baseline, which is audio-only), with a best per-class F1 of 0.88 on Sad. It is the first benchmark of EDSN which is verified independently.