When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model
It is found that applying textual guidance too early can limit its ability to identify answer-relevant visual regions, whereas text-to-visual attention becomes more informative at intermediate decoder depths.