Skip to content

Stable initialization without the CLT

Sep 2026 · arXiv.org · Vol abs/2609.30633 · 0 citations · 40 references
Computer Science

TL;DR

This work derives the uniform-phase initialization, which obviates distributional approximation and fully decouples the layers in networks with sine activations, and is the first work to use the sine function's periodic symmetry.

Abstract

Successful training of deep neural networks is highly dependent on the distribution of the initial weights. If the weights are too large, network training blows up; if they are too small, the model fails to learn features. Stable initialization is the optimal moderation between these two extremes. The conventional theory of random networks uses the Central Limit Theorem to control inter-neuron dependencies, which introduces distributional approximation error and coupling between layers. For networks with sine activations, we derive the uniform-phase initialization, which obviates distributional approximation and fully decouples the layers. Ours is the first work to use the sine function's periodic symmetry. Models trained with the uniform-phase initialization outperform the state of the art in neural representation tasks like image and audio fitting. We find that our untuned models are competitive with the best-tuned baselines from previous work and support $\mu$P width scaling.

View source

Similar papers

#machine learning Preprint Oct 2026

Correlational Training of Morphological Neural Networks

Neural networks are typically trained using first-order methods and back-propagation. It is unclear whether this approach is optimal for morphological layers whose weight Jacobians are sparse and whose resulting parameter gradients can be poor. In this work, we propose a novel weight update method for morphological neu...

Konstantinos Fotopoulos, Petros Maragos · 0 citations
Preprint Aug 2026

Learning of deep neural network regression estimates using gradient descent with pruning

Estimation of a regression function from independent and identically distributed data is considered. The $L_2$ error with integration with respect to the design variable is used as the error criterion. An initially randomly pruned fully connected deep neural network with logistic squasher as activation function is fitt...

M. Kohler, V. M. Römer, Adam Krzyżak · 0 citations
#machine learning Preprint Sep 2026

NGN: Learning Neural Network Size as a Differentiable Count

The Neurogenesis Network (NGN), a differentiable parameterization for learning how many ordered structural components a model should use, is introduced and results show that structural capacity can be optimized directly as a count.

Li-Xing Li · 0 citations
#machine learning Preprint Sep 2026

Conditioned Initialization for Attention

This paper proposes conditioned initialization, a principled scheme that initializes attention weights to improve the spectral properties of the attention layer and shows that conditioned initialization can potentially reduce the condition number of the attention Jacobian, leading to more stable optimization.

Hemanth Saratchandran, Simon Lucey · 0 citations
#machine learning Preprint Aug 2026

Residual-Guided Randomized Neural Networks

A simple and broadly applicable residual guided procedure that greedily constructs the hidden layer using a closed form residual decrease criterion and yields a progressive training process with a guaranteed monotonic decrease of the training objective.

M. Akhtar, M. Tanveer, Mohd. Arshad · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.