A Temporal Action Detection Framework Based on Multi-Scale Temporal-Channel Collaboration
TL;DR
A Multi-scale Temporal-Channel Collaboration (MTCC) detection framework based on the Mamba architecture with state space models (SSMs), which efficiently capture long-range temporal dependencies in video sequences and a module designed to extract features from the channel dimension of video.
Abstract
Temporal action detection is a crucial task in the field of video understanding, aiming to localize and recognize the category of actions in untrimmed videos along with their precise start and end times. Despite the success achieved by existing methods, two significant challenges remain: (1) difficulty in modeling long-range dependencies between video segments, leading to inaccurate localization of complex action boundaries and (2) over-reliance on temporal dimension modeling, with insufficient exploration of the representational capabilities within the channel dimension, which limits model performance. To address these challenges, we propose a Multi-scale Temporal-Channel Collaboration (MTCC) detection framework. First, we construct a Long-Range Dependency Enhancement (LRDE) module based on the Mamba architecture with state space models (SSMs), which efficiently capture long-range temporal dependencies in video sequences. Second, leveraging the advantages of multi-scale CNNs, a Multi-Scale Boundary Awareness (MSBA) module is designed to extract local features and enhance the model’s sensitivity to action boundaries. Finally, a Cross-Channel Information Fusion (CCIF) module is designed to extract features from the channel dimension of video. To better integrate features at different scales, a Multi-scale Detection Head (MDH) is employed to dynamically fuse the feature pyramid. Our systematic evaluations on the benchmark datasets THUMOS14 and ActivityNet-1.3 yield impressive results, demonstrating the effectiveness of our proposed method.