Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
TL;DR
Manifold-Constrained Hyper-Connections, a generalisation of residual connections, is investigated as a novel PEFT approach, wrapping frozen OLMo-2 backbones with learned residual routing modules, identifying residual routing as a distinct and promising novel PEFT axis.
Abstract
Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections. This paper investigates Manifold-Constrained Hyper-Connections (mHC), a generalisation of residual connections, as a novel PEFT approach, wrapping frozen OLMo-2 backbones with learned residual routing modules. We find that mHC can finetune frozen Transformers, but that its role differs fundamentally from the original pre-training setting: in finetuning, fixing the residual mixing matrix to identity often improves performance. As a standalone PEFT method, mHC does not consistently outperform LoRA. However, at matched trainable parameter budgets, mHC+LoRA combinations improve language-modelling loss and show task-dependent benchmark gains at both 1B and 7B scale. Overall, our results identify residual routing as a distinct and promising novel PEFT axis.