UvA-DARE (Digital Academic Repository) TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning

AritraBhowmik MohammadMahdiDerakhshani Dennis C. Koelma Martin R. Oswald Yuki M. Asano Cees G. M. Snoek

· 0 citations · 49 references

TL;DR

This paper proposes TWIST, a twin-expert stepwise tuning module that modifies the decoder of the language model using one frozen module pre-trained on image understanding tasks and another learnable one for visual grounding tasks, which allows the MLLM to retain previously learned knowledge and skills, while acquiring what is missing.

Semantic Scholar

View source

Similar papers

Jul 2026

IoUPD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

Experiments on standard referring-expression grounding benchmarks show consistent region-level improvements over strong coordinate-generating baselines, demonstrating that ground-truth boxes can provide useful privileged guidance beyond serving as coordinate labels.

Xiuyuan Zhu, Ke Lu, Hao Wu et al. · arXiv.org · 0 citations

Preprint Aug 2026

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs

This work proposes a training-free hallucination mitigation framework for dynamic, per-instance suppression at test time, and proposes a dynamically combined projection that selectively suppresses the most probable hallucination directions while preserving image-grounded semantics.

Ali Cheraghian, Hamidreza Dastmalchi, Hamed Barzamini et al. · 0 citations

Preprint Aug 2026

Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement

FISA is proposed, a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases that generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion.

Chun-Yang Jiang, Pingping Zhang, Yuzhi Zhao et al. · 0 citations

Preprint Aug 2026

Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models

The Robust-WAM is a general post-training method for video-generation-based WAMs that preserves the VAE-based generative path and adds a lightweight semantic foresight alignment objective on the action stream to retain the large-scale VGM pretraining while grounding actions in appearance-invariant dynamics.

Hao-Dong Yan, Jun-Feng Li, Jun-Jie He et al. · 2 citations

Jul 2026

Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation

This work proposes NOPD (Noisy Student On-Policy Self-Distillation), a simple yet effective self-distillation approach that improves VLMs without any external models or ground-truth answers, and demonstrates that NOPD is a general approach to enhance VLMs, achieving improvements across three models on 12 benchmarks.

Shuai Wang, Daoan Zhang, Zhe Tang et al. · arXiv.org · 0 citations

Preprint Aug 2026

MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations

Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capabil...

Jongsuk Kim, Qi-Yu Wu, Zhuoyuan Mao et al. · 0 citations