Explainable binary gender-aware facial emotion recognition via hybrid deep learning and customized visual dataset

Abstract

Facial emotion recognition (FER) is a critical component in human-computer interaction, but existing systems often suffer from limited demographic awareness and weak interpretability, as most methods overlook the influence of gender on facial expressions and rely on image-level classification without spatial localization. This work proposes an explainable, binary gender-aware FER framework based on a hybrid deep learning-based architecture. We construct FER2025, a curated dataset with 7,386 images and 11,253 bounding-box annotations spanning 12 classes formed by pairing six emotion categories with binary (male/female) gender labels, validated through inter-annotator agreement analysis. YOLOv8 and YOLOv11 are trained as baseline models, and a dual-stream hybrid model, XFDetNet, is proposed to fuse their complementary predictions via Weighted Boxes Fusion, achieving a recall of 94.56%, F1-score of 91.46%, and mAP@50 of 94.50% on the FER2025 test set, outperforming both baselines. Grad-CAM++ is integrated for visual explainability, with insertion/deletion faithfulness metrics confirming that predictions are grounded in semantically meaningful facial regions. Cross-dataset evaluation and computational cost analysis further characterize the framework’s generalization and efficiency. The implementation and related resources are publicly available at: https://github.com/sadman-adib/Explainable-Gender-Aware-Facial-Emotion-Recognition.git.