When Representation Drift Does Not Reflect Functional Contribution: A Regime Analysis of Prompt Tuning

Abstract

Representation similarity metrics are commonly used to validate parameter-efficient fine-tuning (PEFT) methods that adapt large pre-trained language models by updating only a small subset of parameters under the implicit assumption that a large drift implies meaningful adaptation. In the encoder-based classification settings of this study, we demonstrate that this assumption is regime-dependent rather than universal. In Regime~A (frozen encoder, trainable classifier), learned soft prompts exhibit drift comparable to Low-Rank Adaptation (LoRA) yet are entirely dispensable: removing, scaling, or randomizing them changes fewer than 0.3% of predictions (well within the variance of a random seed). A geometric decomposition explains this: 98% of prompt drift lies in the null space of the multi-class classifier weight matrix, computed across all heads and five seeds. In Regime~B (frozen encoder, frozen task-fitted head, trainable prompts), we run two controlled variants: (B1) head taken from a converged Regime~A model and (B2) head re-initialised from a different random seed. We find consistent, statistically significant gains in both, ruling out re-alignment artifacts. In Regime~C, C(LoRA) and C(BitFit) produce representational drift of comparable Centered Kernel Alignment (CKA) magnitude that aligns closely with the decision boundary (orthogonal fraction 0.52--0.55), yielding large performance gains. Three similarity metrics, namely CKA, Singular Vector Canonical Correlation Analysis (SVCCA), and Projection-Weighted Canonical Correlation Analysis (PWCCA), agree on drift magnitude across every regime yet cannot distinguish functional from non-functional adaptation. We do not claim these metrics are generally invalid; rather, we characterise the specific architectural condition under which they become uninformative, that is, the presence of a free trainable component downstream of the drift source.