Tian-Yu He

HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball

Hyperball optimizers constrain parameter norms and update only their directions, establishing a distinct paradigm for neural network optimization. Although this geometry appears fundamentally different from that of conventional Base Optimizers, which update both parameter norms and directions, we show that the two para...