Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
It is found that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training, highlighting the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection.
Minsik Choi, Geewook Kim, Young Geun Kim
· 0 citations