ADAPTIVE CROSS-MODAL ALIGNMENT AND ORTHOGONAL TASK DECOUPLING FOR MULTITASK MULTIMODAL SUMMARIZATION

Abstract

Producing a reliable multimodal summary requires a model to determine which claims are supported across language and vision and then express that evidence coherently. Training only for generation leaves useful cross-modal supervision untapped, yet indiscriminate parameter sharing across auxiliary tasks can merge transferable semantics with task-dependent cues and thereby induce negative transfer. Alignment is made still harder by visually salient background regions that have little bearing on the desired summary. To address these difficulties, we develop a single framework for multimodal summarization, image-text matching, and image-text retrieval. Visual tokens are first weighted through text-conditioned semantic attention; a learned gate then suppresses unhelpful visual responses before fusion. The aligned sequence is decomposed into one common representation and three private, task-oriented representations. Their redundancy is controlled by a Frobenius-norm orthogonality penalty, optimized jointly with the generation, matching, and retrieval objectives. The implementation couples a T5-base language encoder with CLIP ViT-B/16 and maintains 768-dimensional features in both streams. On MSMO, the full system records ROUGE-1, ROUGE-2, ROUGE-L, and BLEU values of 46.95, 22.03, 44.42, and 18.71, respectively, exceeding the single-task and multitask comparators included in the reported evaluation. Removing adaptive alignment, the shared-private decomposition, or the orthogonal term lowers ROUGE-L by 2.91, 2.08, and 0.87 points. Taken together, the findings indicate that text-directed visual filtering and an explicit division between reusable and task-specific information are both consequential for grounded summary generation.