Skip to content
Book Open access

APT: Alignment-Preserving Video Tokenizer for Non-Destructive Image-to-Video Adaptation

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 74 references

Abstract

Adapting Image Large Language Models (Image-LLMs) to long-form video is hindered by the large number of redundant visual tokens. Existing video adapters reduce this cost, but their unconstrained projections may generate representations that deviate from the input distribution expected by a frozen Image-LLM, degrading its pre-aligned visual--language semantics. We introduce the Alignment-Preserving video Tokenizer (APT), a FLOP-efficient interface for non-destructive Image-to-Video adaptation. APT first identifies informative spatio-temporal anchors with a Spatio-Temporal Visual Token Refiner, and then condenses them using a Text-Conditioned Token Aggregator. The final value aggregation is constrained to a parameter-free convex combination of the selected anchors, providing a structural inductive bias that conditionally limits representational drift without requiring external shot detection or additional image training data. APT generalizes across diverse Image-LLM backbones and achieves leading performance on short-, long-, and open-ended video understanding benchmarks while substantially reducing adapter FLOPs and inference latency. Evaluations on static-image benchmarks further show statistically insignificant performance variations, indicating that APT mitigates adapter-induced distribution shift while preserving zero-shot compatibility with the original Image-LLM.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.