Output-aware Residual Stream Pruning for Large Language Models
This work characterize the effect of a residual-stream perturbation through both its activation covariance and the local sensitivity of the model output, and derives a tractable spectral upper bound that reduces subspace selection to an eigendecomposition of a sensitivity-weighted covariance matrix, retaining the effic...