Integrating Domain Knowledge for Robust and Interpretable Deep Neural Networks in Facial Affect Recognition

Abstract

Facial affect recognition is a key component of human-centered AI, enabling systems to respond appropriately to human nonverbal signals. The deployment of deep learning for facial affect recognition is critically hindered by vulnerability to data-driven biases and a lack of transparency. This thesis addresses these challenges through a multi-faceted framework, structured along the CRISP-ML(Q) machine learning lifecycle, integrating domain knowledge to build more robust and interpretable systems. First, it quantifies demographic bias through extensive subject-level metadata annotation, uncovering performance gaps across demographic groups in leading facial expression recognition (FER) classifiers. To mitigate demographic and multi-label class imbalance, an optimization-based data balancing technique, MLOPT, is introduced to improve performance, fairness, and generalization. Second, the training of deep learning models is guided by injecting anatomical knowledge about facial Action Units (AUs), that is, localized facial muscle activations defined by the Facial Action Coding System (FACS). This is achieved through the CorrLoss regularization term and a Facial Movement Constraint Network (FMC-Net), which together improve model robustness and predictive performance. Third, interpretability is enhanced through a fuzzy-logic ensemble that combines expert rules with model predictions. This dual approach enables self-validation of the models by flagging uncertain outputs while providing inherently interpretable behavior through logical rules. Finally, a new evaluation framework, RelEx, is proposed to assess the physiological plausibility of trained models by quantifying the alignment between feature attributions and expert-defined facial regions. By addressing these challenges with targeted solutions, this research demonstrates that systematically integrating domain knowledge moves FER beyond performance optimization. The resulting framework provides a foundation for developing AI systems that are not only high-performing but also robust and interpretable.