Skip to content

Derivative-Free Structured Updates for Muon

Sep 2026 · 0 citations · 15 references
Mathematics Computer Science

TL;DR

A derivative-free framework is developed that constructs Muon-style updates from structured finite differences from structured finite differences, and shows that random rank-one probing can reduce the number of function evaluations substantially, at the cost of less accurate updates.

Abstract

Muon updates matrix-valued neural-network parameters by orthogonalizing a gradient-based momentum matrix. Its reliance on derivatives limits its use when gradients are unavailable or unreliable. We develop a derivative-free framework that constructs Muon-style updates from structured finite differences. Four variants are considered: full entrywise recovery, random low-rank surrogates, basis-aligned rank-one probing, and direct structured search. Exhaustive basis-aligned probing is equivalent, up to positive scaling before ideal polar orthogonalization, to coordinate finite differences. Matrix-regression experiments show that random rank-one probing can reduce the number of function evaluations substantially, at the cost of less accurate updates. Controlled noisy-gradient experiments on regression and a neural network illustrate when accurate function values can compensate for an unreliable gradient oracle. A small CartPole study further examines orthogonal rank-one probes under a fixed episode budget. These results support structured probing as a practical option for selected black-box problems; they do not establish a general convergence guarantee or an advantage over accurate, inexpensive gradients.

View source

Similar papers

Muon Under Gradient Noise and the Limits of Orthogonalization Near Optima

Muon replaces the momentum buffer of each weight matrix by its orthogonal polar factor. We ask what this orthogonalization does near an optimum, where minibatch noise dominates the gradient. Under a Gaussian noise model, the expected Muon update becomes a scaled gradient step, so a linear method with a suitably matched...

Xiao-Hui Xie · 0 citations
Preprint Sep 2026

Gradient-Free Optimization for Matrix functions

An alternative to the standard random gradient estimator is introduced, allowing for the projection step of spectral descent to be done at no extra cost and it is shown that by exploiting this low-rank property one obtains much faster convergence to good approximate solutions.

S. Allen, Cash Cherry, Aidan Eck et al. · 0 citations
Preprint Aug 2026

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

This work introduces RODE, which gives the radial and directional components separate update rules and step sizes in the matrix Frobenius norm, and suggests that decoupling radial and directional dynamics offers a more effective and controllable approach to matrix optimization.

Guo-Xiang Xu, Bince Qu, Qi Sun et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Training-Loss Guarantees for Muon with Finite-Step Newton--Schulz Orthogonalization

Existing convergence analyses of Muon either assume exact orthogonalization or analyze classical Newton--Schulz polynomials, and guarantee only stationarity, so it is unresolved what Muon's five tuned Newton--Schulz steps preserve and whether that suffices to reach a prescribed neural-network training loss. We establis...

Amartya Roy, Souvik Chakraborty · 1 citation
#artificial intelligence Preprint Sep 2026

Beyond the Matrix Sign: Quadratic Spectral Descent

Muon emerges as a strong competitor of the AdamW for LLM pretraining, because the matrix-wise update it employs can potentially incur smaller second-order penalty than the once dominating AdamW, which performs coordinate-wise update. However, the spectral flattening procedure in Muon is quite debatable since it discard...

Qiao-Zhe Zhang, Jun Sun, Ying-Zhuang Liu · 2 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.