Skip to content

Convergence of Practical Muon

Sep 2026 · arXiv.org · Vol abs/2609.33152 · 2 citations · 53 references
Computer Science Mathematics

TL;DR

This paper interprets practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted $\ell_2$ regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective and establishes the first convergence guarantee for practical Muon in the stochastic nonconvex setting.

Abstract

Muon is emerging as a promising alternative to AdamW for large-scale neural network training, yet theoretical understanding of its practical implementation remains incomplete, as existing analyses often simplify or omit two key components: (i) practical Newton--Schulz iterations with empirically tuned polynomial coefficients $(3.4445,-4.7750,2.0315)$; and (ii) decoupled weight decay for regularization. In this paper, we provide an optimization interpretation and establish convergence for practical Muon, jointly accounting for both components. Specifically, we interpret practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted $\ell_2$ regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective. We then establish, to our best knowledge, the first convergence guarantee for practical Muon in the stochastic nonconvex setting, with an $\mathcal{O}(T^{-1/4})$ convergence rate in terms of the expected Frobenius norm of the gradient, improving the dimension dependence of the best known AdamW's convergence rate by a factor of $\sqrt{d}$, where $T$ is the iteration horizon and $d$ is the parameter dimension. Experiments further support the theoretical convergence results.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization

Existing convergence analyses of Muon either assume exact orthogonalization or analyze classical Newton--Schulz polynomials, and guarantee only stationarity, so it is unresolved what Muon's five tuned Newton--Schulz steps preserve and whether that suffices to reach a prescribed neural-network training loss. We establis...

Amartya Roy, Souvik Chakraborty · 1 citation
#machine learning Preprint Oct 2026

Spectrally Targeted Muon

Spectrally Targeted Muon is introduced, which orthogonalizes only the singular values above or below a threshold $\tau$ so that varying $\tau$ interpolates between normalized SGD and Muon, and is evaluated on the CIFAR-10 and NanoGPT speedruns.

Vishrut Goyal, R. Ram Kumar · 1 citation

Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum

Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Nesterov correction. We analyze these layer-wise finite-step updates jointly on a coupled nonconvex objective, rather than replacing them by exact polar factors or one glo...

Han-Yang Peng, Hui Wang, Yue Yu · 2 citations
Preprint Oct 2026

Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations

Muon has demonstrated strong empirical performance in large-scale language-model training, motivating a deeper theoretical understanding of its practical implementation. Unlike the idealized polar-factor update, practical Muon employs a fixed number of quintic Newton-Schulz iterations. Existing analyses primarily estab...

Ping-Qi Li, Vincent Y. F. Tan · 1 citation
#machine learning Preprint Oct 2026

When and How Does Muon Precondition? The Role of Structured Gradient Noise

Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient noise geometry shapes Muon's preconditioning benefits. We first show, theoretically and empirically, that under the classical isotropic noise assumption, M...

Xiao-Ran Chen, Bing-Hui Li, Li-Wei Wang et al. · 0 citations

Convergence guarantees for Muon: New parameter regimes and generalizations

In this paper, we establish the first asymptotic convergence guarantees for the Muon algorithm through a more accurate proxy for the Newton-Schultz iteration than the typical matrix sign function. We prove that, for appropriate choices of hyperparameters, the iterates satisfy $\lim_{k\to\infty}\|\nabla f(x_k)\|=0$, and...

Arthur Castello Branco de Oliveira, Dhruv D. Jatkar, G. S. Vicinansa et al. · 4 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.