Target-Token Asymmetry Can Masquerade As Depth Specialization
Layer-ablation studies often perturb early or late layers of a language model, compare the damage to prediction on two corpora, and read a larger early-layer effect on one corpus as early-layer specialization for its task. Such a comparison averages over the tokens that each corpus asks the model to predict, and these...
Dan-Lin Li, Miao Li, Xu Han et al.
· 0 citations