Explainable Multimodal Depression Detection via Cross-Platform Transformer Fusion

Abstract

Depression is a globally prevalent mental health disorder that is increasingly investigated through digital behavioral data derived from social media platforms. However, most existing detection models relies on using single-modality or single-platform datasets, which limit their robustness and generalizability in real-world environments which can consist of two or more platforms. This study proposes an explainable cross-platform depression detection framework that integrates textual data from Reddit with audiovisual data from YouTube using Transformer-based architectures. The framework is evaluated on heterogeneous real-world data consisting of 3,553 text posts from Reddit and 1,823 audiovisual vlogs from the LMVD dataset. Two multimodal fusion strategies, early fusion and late fusion, are implemented then evaluated against a text-only baseline. Experimental results indicate that the late fusion strategy achieves the most balanced performance, attaining an accuracy of 80.55% and an F1-score of 80.86%, thereby outperforming both the early fusion approach and the unimodal model. Furthermore, explainability analyses based on SHAP and Integrated Gradients demonstrate that visual and audio cues contribute at different temporal stages of the prediction process, with visual cues exerting greater influence during earlier segments and audio cues becoming more prominent near peak risk regions. These findings suggest that decision-level multimodal integration, combined with transparency mechanisms, provides a robust and interpretable framework for depression screening across heterogeneous social media platforms.