Self-Distillation as a Structure-Dependent Mechanism in Multimodal Dialogue via Multi-Context Modeling

Abstract

Multimodal dialogue understanding requires integrating heterogeneous signals such as language, audio, and visual cues while capturing complex contextual dynamics in multi-party interactions. Although recent models achieve strong performance, they typically rely on unified representations and treat self-distillation as a generic enhancement, leaving its role in structured dialogue modeling unclear. We investigate self-distillation as a structure-dependent mechanism whose effectiveness depends on how conversational context is represented. To this end, we propose a multi-context framework that explicitly separates dialogue context into global interaction dynamics and speaker-specific characteristics. Experiments on a multi-party dialogue dataset show that (1) the effectiveness of self-distillation varies across model configurations, (2) its effectiveness is influenced by teacher signal design, and (3) the best overall performance is achieved when self-distillation is combined with structured multi-context representations. These findings suggest that performance gains often attributed to self-distillation are better explained by representation structure, highlighting the importance of jointly designing contextual representations and self-distillation strategies.