Skip to content

Convergence guarantees for Muon: New parameter regimes and generalizations

Sep 2026 · arXiv.org · Vol abs/2609.30546 · 4 citations · 18 references
Mathematics Computer Science

Abstract

In this paper, we establish the first asymptotic convergence guarantees for the Muon algorithm through a more accurate proxy for the Newton-Schultz iteration than the typical matrix sign function. We prove that, for appropriate choices of hyperparameters, the iterates satisfy $\lim_{k\to\infty}\|\nabla f(x_k)\|=0$, and, under a global Polyak-\L{}ojasiewicz condition, that the sequence of function values converges linearly. The key insight is that the regularization, implicit in Muon's Newton-Schulz implementation, induces a bounded preconditioner, exposing Muon as a \emph{preconditioned Polyak heavy-ball} method and enabling a classical Lyapunov analysis. This observation naturally motivates applying the same preconditioning structure to the Nesterov gradient evaluation. We formalize this idea by introducing \emph{Muesterov}, a Nesterov-based variant of Muon, and prove that it enjoys the same convergence guarantees, extending the theoretical framework beyond the heavy-ball setting. Numerical experiments on a scalar cross-entropy problem corroborate the theory and illuminate the joint role of the learning rate and the Newton-Schulz regularizer in controlling convergence. Preliminary numerical simulations training the nanoGPT dataset provide intuition regarding the relevance of the observations in this paper to practical applications.

View source

Similar papers

Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum

Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Nesterov correction. We analyze these layer-wise finite-step updates jointly on a coupled nonconvex objective, rather than replacing them by exact polar factors or one glo...

Han-Yang Peng, Hui Wang, Yue Yu · 2 citations
Preprint Oct 2026

Quantitative $\Gamma$-Convergence: A Variational Extrapolation Framework

This paper introduces a new extrapolation framework within the Calculus of Variations. We develop a robust quantitative theory of the $\Gamma$-convergence approach of De Giorgi by combining it with Gagliardo's work on relative completions in interpolation theory. A central new tool is the extrapolation of \emph{variati...

Oscar Domínguez, M. Milman · 0 citations
Preprint Aug 2026

Convergence rates for the extreme value theorem via Stein's method

We derive convergence rates for the approximation of the Fr\'echet distribution $\mathcal{F}(\alpha)$ with parameter $\alpha>0$ by sequences of renormalized maxima in the extreme value theorem. Our proofs rely on the application of the infinitesimal generator approach to Stein's method to max-stable distributions, usin...

B. Costacèque, Nicolas Privault · 0 citations

Convergence of Practical Muon

This paper interprets practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted $\ell_2$ regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective and establishes the first convergence guarantee for practical Muon in th...

Hao-Nan Wang, Yu Wu, Minghui Liwang et al. · 2 citations
Preprint Aug 2026

Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimization

Muon has emerged as a strong optimizer for the matrix-valued parameters in large language model pretraining, approximately orthogonalizing its momentum with a few Newton-Schulz iterations. Existing theory either replaces this iteration with the exact polar factor it approximates, or treats its finite depth as an approx...

Ming-Yi Li, Taira Tsuchiya · 5 citations · ⚡1
#machine learning Preprint Oct 2026

Last-Iterate Convergence Rate of Normalized Gradient Descent under H\"older Smoothness

Normalized gradient descent is a widely studied adaptive optimization method. Most existing analyses focus on the best iterate or a weighted average of the iterates, whereas practical implementations typically return the last iterate. In this paper, we study the last-iterate convergence of normalized gradient descent f...

Yuki Takezawa, Eduard A. Gorbunov · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.