Video-text temporal localization requires precise alignment between natural language queries and corresponding video segments, a fundamental challenge in multimodal understanding. We present a novel framework that addresses two critical limitations of existing methods: inadequate modeling of hierarchical temporal structure and inability to handle complex many-to-many correspondences between modalities. Our approach introduces a multi-scale temporal convolutional encoder that captures motion patterns across different temporal granularities - from instantaneous frame transitions to extended action sequences. We further propose a capsule-based dynamic routing mechanism that iteratively refines segment-query associations through structured agreement updates, enabling flexible modeling of non-monotonic alignments. These components are unified through a multi-task learning objective that jointly optimizes temporal boundary regression, cross-modal semantic alignment, and capsule diversity. Extensive experiments on ActivityNet Captions demonstrate significant improvements, achieving 42.9% Recall@0.5 and 41.1% mean IoU, surpassing strong transformer-based baselines while maintaining computational efficiency. Our results validate that combining hierarchical temporal modeling with structured semantic routing provides an effective solution for fine-grained video-language understanding.
Gengtian Shi, Jinze Yu, Chenhao Wu et al.· 0 citations
Action grounding in videos aims to identify precise temporal boundaries in untrimmed videos that correspond to natural language queries, a capability essential for vision–language–action systems and embodied intelligence. Existing methods suffer from two limitations: treating all temporal scales uniformly through global attention, and assuming simplistic one-to-one correspondences between video frames and text tokens. To address these issues, we propose a cross-modal temporal alignment framework that combines a multi-scale temporal convolutional encoder with capsule-based dynamic routing. The multi-scale encoder applies parallel 1D convolutions with varying kernel sizes to capture hierarchical motion patterns at $O(T)$ complexity, enabling efficient modeling of both instantaneous transitions and extended action sequences. The capsule-based routing mechanism iteratively refines video–text alignments through agreement-based coefficient updates, producing sparse and structured many-to-many correspondences between video frames and text tokens. These two components are unified through a multi-task objective jointly optimizing temporal boundary prediction, cross-modal semantic alignment, and capsule diversity. Experiments on ActivityNet Captions demonstrate that our framework achieves 42.9% Recall@0.5 and 41.1% mean IoU while maintaining linear-time efficiency for long-video processing.
Gengtian Shi, Chenhao Wu, Shaofei Wang et al.· IEEE Access· 0 citations