Debiasing Temporal Sentence Grounding via Semantic-Aware Multimodal Query Reconstruction

Abstract

Temporal Sentence Grounding in Videos (TSGV) aims to identify the temporal segment in a video that semantically corresponds to a given textual query. Although effective semantic alignment across modalities is crucial for this task, existing TSGV benchmarks exhibit annotation biases such as an over-reliance on specific words or temporal positions. These biases often lead models to learn spurious correlations rather than genuine cross-modal semantics, resulting in poor generalization to out-of-distribution (OOD) scenarios. To address this challenge, we propose a reconstruction-based debiasing framework. By requiring the model to infer masked query words from both visual and textual context, our reconstruction objective breaks the reliance on superficial temporal priors and encourages genuine cross-modal understanding. Its masking follows a Part-of-Speech (POS)-aware strategy that prioritizes semantically significant content words over trivial linguistic cues, keeping the reconstruction task both challenging and meaningful. Beyond reconstruction, we introduce a cross-segment push loss that aligns the representations of query–segment pairs within a video according to their temporal overlap, making them segment-discriminative and reducing positional bias at the representation level. Because the proposed method is applied only during training, the model incurs no additional computational overhead at inference. Experimental results show that the proposed method achieves consistent improvements in OOD generalization on both the Charades-CD and ActivityNet-CD datasets. Furthermore, in-depth ablation studies validate the specific contribution of each component to the debiasing performance. These findings confirm the effectiveness of our framework in enhancing multimodal semantic alignment for TSGV.