Cross-Modal Temporal Alignment for Action Grounding in Videos

Abstract

Action grounding in videos aims to identify precise temporal boundaries in untrimmed videos that correspond to natural language queries, a capability essential for vision–language–action systems and embodied intelligence. Existing methods suffer from two limitations: treating all temporal scales uniformly through global attention, and assuming simplistic one-to-one correspondences between video frames and text tokens. To address these issues, we propose a cross-modal temporal alignment framework that combines a multi-scale temporal convolutional encoder with capsule-based dynamic routing. The multi-scale encoder applies parallel 1D convolutions with varying kernel sizes to capture hierarchical motion patterns at $O(T)$ complexity, enabling efficient modeling of both instantaneous transitions and extended action sequences. The capsule-based routing mechanism iteratively refines video–text alignments through agreement-based coefficient updates, producing sparse and structured many-to-many correspondences between video frames and text tokens. These two components are unified through a multi-task objective jointly optimizing temporal boundary prediction, cross-modal semantic alignment, and capsule diversity. Experiments on ActivityNet Captions demonstrate that our framework achieves 42.9% Recall@0.5 and 41.1% mean IoU while maintaining linear-time efficiency for long-video processing.