Preprint

AdaLoop: Thinking with the Vision Encoder

Xinran Nicole Han Yi-Yang Lu Katelyn Gan Todd E. Zickler

Oct 2026 · 0 citations · 83 references

Computer Science

TL;DR

AdaLoop is presented, a framework for question-conditioned visual recurrence in VLMs that adapts visual computation to the question and reuses the encoder's transformations across multiple iterations, with the pretrained vision weights frozen.

Abstract

Most vision-language models (VLMs) treat the vision encoder as a static feature extractor: it encodes an image once, independently of the question, and leaves subsequent reasoning to the language model. Yet different questions may require different visual operations and processing depths. We present AdaLoop, a framework for question-conditioned visual recurrence in VLMs. AdaLoop adapts visual computation to the question and reuses the encoder's transformations across multiple iterations, with the pretrained vision weights frozen. We also introduce VisRO, a procedural dataset and generation pipeline for training and evaluating vision-centric reasoning. On six VisRO tasks, AdaLoop outperforms standard fine-tuning and question-conditioned single-pass baselines with matched training data and optimizer updates. These gains hold across three VLM families on both in-distribution and out-of-distribution examples. Our findings suggest that pretrained vision encoders can contribute to reasoning through task-directed, iterative visual computation.

arXiv Semantic Scholar

View source

Similar papers

Preprint Sep 2026

ConvCue: Complementary Visual Inductive Biases for Vision-Language Models

Results show that pretrained convolutional representations, when integrated through learned adaptation and fusion, can improve the visual understanding of existing VLMs without replacing their original visual encoders.

Zi-Xuan Lan, Shi-Chu Sun · 0 citations

#artificial intelligence Preprint Oct 2026

VICO: Visual Environments Co-Evolving for Vision-Language Model Reasoning

Reinforcement learning with verifiable rewards (RLVR) has become a standard recipe for post-training vision-language models (VLMs), but it typically assumes a static training environment. As the actor improves, fixed tasks drift out of its learning frontier: many become trivial, others remain unsolvable; and the learni...

Meng Lu, Li-Geng Zhu, Olivia C. Xiao et al. · 1 citation

Preprint Sep 2026

Decoupling Vision, Language, and Action for Efficient Multi-Task Robot Policies

Within this trained-task regime, DEM sits on the observed success--latency--energy frontier and provides a strong, efficient baseline for language-conditioned robot skills.

Xia-Tao Sun, Chen Liang, Zi-Yao Zeng et al. · 2 citations

#artificial intelligence Preprint Sep 2026

VoT: Vision-of-Thought for Unified Multimodal Representation Alignment

Vision-of-Thought (VoT), a framework that introduces a discrete visual-thinking layer between vision-language models (VLMs) and diffusion transformers (DiTs) that improves semantic alignment and provides a structured interface for interpretable and controllable generation is proposed.

Jingxiang Sun, Chao Liao, Zheng-Xiong Luo et al. · 0 citations

#computer vision Preprint Aug 2026

Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models

An embodied scene question-answering (QA) interface in which vision never enters the language model is studied, showing the gain survives paraphrase, attributing it to decision-aligned computation rather than answer-string leakage, while novel judgment vocabularies bound its scope.

Cong Xu, Ravi Sankar · 1 citation

#machine learning Preprint Oct 2026

From Patching to Pruning Visual Computation in Vision Language Models

Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every decoder layer, even when token-specific visual computation is unnecessary at many depths. We introduce Patch-to-Prune (P2P), inspired by Mechanistic Interpretability, a tr...

Rahul Chowdhury, Timothy A. Rupprecht, Xuan Shen et al. · 0 citations