A Long-sequence Hierarchical Multimodal Fusion Framework for Personality-aware Depression Detection

Abstract

Depression detection in older adults is an important human-aware multimedia problem, where latent mental-health states must be inferred from non-invasive behavioral and individual-difference cues. This task is challenging because depressive symptoms are often subtle, heterogeneous, and reflected across multiple channels, including speech, facial behavior, language use, and personality-related differences. The MPDD-AVG Elder dataset provides a personality-aware multimodal setting for elderly depression detection, but it also introduces practical challenges such as long and variable-length audio-visual sequences, limited labeled data, and the underuse of textual semantics embedded in audio. To address these challenges, we propose a long-sequence hierarchical multimodal fusion framework for subject-level depression prediction. Our method models each subject through a chunk-sample-subject hierarchy. Long audio-visual sequences are divided into MFCC-aligned chunks, encoded with an audio-visual interaction module, and aggregated by attention pooling across chunks and samples. In addition, we introduce ASR-derived textual representations to capture semantic and self-description cues from interview responses. For prediction, we combine ordinal classification with PHQ-9 regression and we further apply LLM-based PHQ-9 score calibration to improve PHQ-9 prediction stability under the small-data setting. Experiments on the MPDD-AVG Elder dataset demonstrate the effectiveness of the proposed framework for elderly depression detection.