Diffract: Spectral View of LLM Domain Adaptation
It is found that CPT leaves singular value spectra largely invariant, with adaptation driven mainly by changes in singular vectors, and domain connectivity is identified - linear interpolation between CPT checkpoints yields smooth domain-quality interpolation without notable degradation on either domain.