A practical training recipe for the normalized Transformer and its evaluation on modern hybrid Mamba-2--Transformer Mixture-of-Experts models shows that the 30B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens.
Abstract
The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this paper, we describe a practical training recipe for nGPT and evaluate it on modern hybrid Mamba-2--Transformer Mixture-of-Experts (MoE) models. The recipe introduces Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms. Compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 30B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens. The recipe scales across the models considered, which contain up to 30B total parameters.
The Neurogenesis Network (NGN), a differentiable parameterization for learning how many ordered structural components a model should use, is introduced and results show that structural capacity can be optimized directly as a count.
We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models. In particular, we focus on the gap between the theoretical scaling rules derived for SSMs under zero-order-hold (ZOH) discretization at inf...
Jimin Seo, Gyubok Lee, Yeonsik Jo et al.· 0 citations
Neural networks are typically trained using first-order methods and back-propagation. It is unclear whether this approach is optimal for morphological layers whose weight Jacobians are sparse and whose resulting parameter gradients can be poor. In this work, we propose a novel weight update method for morphological neu...
This work introduces Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models, and shows that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately unif...
Feng-Zhuo Zhang, Shu-Che Wang, Sheng-Gui Li et al.· 0 citations
This work evaluates the subspace Levenberg-Marquardt algorithms for regression and classification tasks in neural networks and compares the performance of subspace LM variants with the classical LM method, as well as other popular first-order algorithms, such as stochastic gradient descent and Adam.
LA-ReduNet is proposed, a lightweight adaptive architecture that refines the layer-wise update rule and enables discriminative feature representations to be obtained with substantially fewer unfolded layers, thereby reducing by an order of magnitude the number of layers required for the MCR$^2 objective to reach a stab...
Zhenglin Huang, Qi-Fa Yan, Bin Dai et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.
