Micro-video Scene Classification Method Based on Generative Feature Disentanglement and Multimodal Dynamical Fusion

Abstract

Since micro-video is typically captured in complex real-world environments, it often contains a large amount of noise and redundant information. They can weaken the expression of key semantics, thereby affecting the model's accuracy in scene recognition and classification. Owing to the production, editing, or other influencing factors of micro-video content, the correlations among different modalities exhibit uncertainty, and the semantic strength of each modality is often inconsistent and affected by noise. These issues pose challenges to multimodal fusion, including uncertainties in the fusion weights and noise levels of each modality, as well as semantic differences across different modalities. To address the above problems, this paper proposes a Micro-video Scene Classification Method based on Generative Feature Disentanglement and Multimodal Dynamical Fusion (MDF-GFD). In the Generative Feature Disentanglement module, we tackle this problem from a data-generation perspective, decomposing each modality's data into modality‑specific, modality‑shared, and noise components. By minimizing the generative loss, we disentangle modality‑specific and modality‑shared representations from raw data and leverage them jointly for discriminative learning within individual modalities. In the Dynamic Multimodal Fusion module, the modality-specific and modality-shared components of each modality are integrated, and multimodal consistency and complementarity are effectively combined through an information-entropy-based dynamic fusion mechanism, which adaptively fuses modalities according to their information contribution. Experimental results demonstrate the effectiveness of the proposed MDF-GFD method in micro-video scene classification.