Locking Pretrained Weights via Deep Low-Rank Residual Distillation
This work proposes DLR-Lock, a method where the purveyor of the model purposely replaces each pretrained MLP in their model with a deep low-rank residual network of comparable parameter count, forcing activation memory that grows linearly with depth during backpropagation, and exploits the inference-training asymmetry...