Extending the scale generalization of the Vision Transformer without fine-tuning.

Abstract

The "train low, deploy high" paradigm offers significant practical advantages by minimizing training overhead while enabling high-fidelity inference through increased spatial resolutions. However, Vision Transformers (ViTs) often suffer from poor zero-shot generalization to unseen resolutions compared to their convolutional counterparts. We attribute this deficiency to two fundamental phenomena: intra-patch spectral drift, where image resizing suppresses discriminative mid-to-high frequency components due to interpolation-induced low-pass filtering, and inter-patch positional awareness collapse, where the interpolation of absolute position embeddings distorts spatial priors and causes the effective receptive field to degenerate into isolated patches at larger scales. To mitigate these issues, we propose the Multi-Scale Vision Transformer (MSViT), which integrates Spectral-Constrained Convolution for adaptive frequency-weighted patch embedding, Horizontal-Vertical Separable Attention to enforce a full-span cross-shaped effective receptive field, and Reparameterized Convolutional Position Embedding to provide boundary-aware spatial bias without the need for interpolation. When trained exclusively at 224 × 224, MSViT demonstrates remarkable robustness across a broad range of test resolutions, maintaining consistent and stable accuracy as the input scales from 128 × 128 up to 640 × 640. Our work underscores that explicit modeling of spectral stability and spatial structure is essential for developing resolution-flexible vision transformers.