Jul 2026
Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates
A diagnostic optimizer, Shape--Boundary Orthogonal AdamW (SBO-AdamW), is implemented that optimizes g_W-g_b\mu^\top and g_b with independent Adam states and compensates the weight-induced boundary displacement and a stable centered-affine parameterization is identified as the required next step.
Gongyue Zhang, Yi-Xuan Sheng, Donghan Liu et al.
· arXiv.org · 0 citations