Test-Time Logit Prompting (TLP), a lightweight source-free test-time adaptation framework for visual recognition with missing modalities, is proposed, which optimizes logit prompts with uncertainty-aware adjustment and modality-complete consistency regularization, adaptively adjusting prediction confidence while preserving semantic consistency.
Abstract
Vision-language models (VLMs) have achieved remarkable performance by leveraging complementary information from large-scale image-text pairs. However, missing-modality inputs are commonly encountered during real-world deployment, often leading to significant performance degradation. Existing methods primarily enhance model robustness by learning modality compensation strategies from source training data. However, their reliance on source training data makes them difficult to apply when original data are unavailable due to privacy, storage, or accessibility constraints, such as clinical applications and personalized AI services. This raises an important yet underexplored question: can VLMs be efficiently adapted at test time for visual recognition with missing modalities without accessing source training data? To this end, we propose Test-Time Logit Prompting (TLP), a lightweight source-free test-time adaptation framework for visual recognition with missing modalities. To address missing-induced prediction shifts, TLP optimizes logit prompts with uncertainty-aware adjustment and modality-complete consistency regularization, adaptively adjusting prediction confidence while preserving semantic consistency. Extensive experiments across diverse vision-language benchmarks demonstrate that TLP consistently enhances recognition performance under missing-modality scenarios, achieving up to 8\% improvements while requiring only hundreds of tunable parameters and a few test-time optimization steps.
PuRF is introduced, a novel PuRiFication-driven cache-based method for multi-label test-time adaptation of vision-language models that consistently outperforms state-of-the-art methods on ViT-B/32 across five datasets.
Yiwen Liang, Hui Chen, Yizhe Xiong et al.· 0 citations
This work proposes CARAT, which decouples model reliance from runtime corruption detection to guide omission or attenuation, amortizing reliance estimation through source training and position CARAT as a practical default to wearable TTA, offering competitive robustness with modest computational requirements and benefi...
Payal Mohapatra, Yue-Yuan Sui, Hao Yang et al.· 0 citations
The proposed TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation, combines image and question dependence with calibrated confidence to construct a response-level reward that favors jointly grounded responses.
Xinrui He, Ting-Wei Li, Jun-Ting Wang et al.· 0 citations
Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness, is proposed and suggested as a practical route for robustness correction.
Song-Yuan Sui, Zhen Tan, Mohan Zhang et al.· 0 citations
ECHO-$k$ is introduced, a task-agnostic and self-supervised learning principle for modality acquisition that uses a deep model's internal pretrained representations as proxy targets that summarize cross-modal information and consistently improves budgeted downstream performance across diverse foundation-model backends.
Eeshaan Jain, L. Bleistein, B. Deplancke et al.· 0 citations
This work proposes FourIer Representation Editor (FIRE), a novel multimodal TTA approach that directly edits semantically rich intermediate representations, and demonstrates the superiority of FIRE over existing multimodal TTA methods.
Long-Fei Huang, Xiang-Yu Wu, Yang Yang· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.