TextFilter: Filtering Semantic-Temporal Representations with Texts for Video Action Recognition
Abstract
Vision–Language Models (VLMs) have advanced video action recognition, yet most existing approaches still rely on text as a static semantic cue. In real video data, however, the primary challenge often lies in the noise (dominant backgrounds, scene biases, and temporally irrelevant motions) that obscure discriminative patterns. We build on the mixture-of-experts framework and propose TextFilter, which repositions language from a passive descriptive signal to an active mechanism for semantic-temporal filtering. Rather than solely injecting semantics, TextFilter uses text to suppress misleading visual signals. A text-guided background attenuation module first reduces semantic interference by suppressing background signals. In parallel, text-driven semantic and temporal Mixture of Cross-modal Experts (MoCE) modules leverage fine-grained LLM-generated descriptions to guide expert selection. The semantic branch focuses on object-relevant regions, while the temporal branch emphasizes action-critical motion phases. Together, these components yield cleaner representations, more reliable routing, and stronger video–text alignment. We assess TextFilter across five widely used benchmarks: Kinetics-400/600, UCF-101, HMDB-51, and SSv2. Our TextFilter delivers competitive performance compared to state-of-the-art methods in zero-shot, few-shot, and fully supervised regimes, while remaining highly parameter-efficient.