Aug 2026· arXiv.org· Vol abs/2608.10374· 0 citations· 22 references
Computer Science
TL;DR
Fisher8 is derived, an output-layer gradient correction that reorients and rescales updates using Fisher geometry rather than Euclidean geometry, and it is shown that prior stabilizers converge on overlapping components of this geometric correction.
Abstract
Training neural networks to jointly predict mean and uncertainty estimates from noisy observations can be unstable, prompting a series of independent stabilization efforts. We argue that these interventions highlight a common underlying issue where gradient steps are poorly aligned with the geometry of the loss landscape. To better align updates with local curvature, we derive Fisher8, an output-layer gradient correction that reorients and rescales updates using Fisher geometry rather than Euclidean geometry. Unlike past stabilizers, Fisher8 introduces no data-dependent hyperparameters beyond learning rate and admits an approximate KL trust radius between successive predictive distributions. We show that prior stabilizers converge on overlapping components of this geometric correction. Across multidimensional regression and representation-learning tasks, Fisher8 obtains superior likelihood--error tradeoffs, predicts calibrated uncertainty estimates, and learns rich uncertainty-aware feature spaces.
Quadratic approximations provide a common lens for studying neural network optimization, but recent evidence challenges their predictive validity. In full-batch gradient descent with learning rate (LR) $\eta$, Cohen et al. (2021) observed the Edge of Stability (EoS), where the largest Hessian eigenvalue concentrates ne...
Yu-Hang Cai, Hao-Dong Wen, Hao-Feng Huang et al.· 2 citations
Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training...
Optimal Transport Dropout (OTD) is introduced, which offers a way to learn structured predictive uncertainty without explicit posterior inference or ensembles of independently trained predictors.
Giacomo Lorenzon, Francesco Regazzoni· arXiv.org· 0 citations
Recent successes of matrix- and curvature-based optimizers have renewed interest in how update geometry shapes learning. These methods normalize or precondition updates, changing how different components progress during training. In deep matrix factorization, the geometry that slows gradient descent (GD) favors low-ran...
Physics-informed Gaussian processes (GPs) provide a principled framework for scientific inference by expressing uncertainty over empirical measurements, boundary conditions, and governing equations. However, they struggle to scale to larger datasets and nonlinear information. Conversely, diffusion models excel at learn...
Tobias Weber, Carla Sagebiel, Philipp Hennig· 0 citations
Modern deep learning architectures excel at learning complex representations from high-dimensional data, achieving remarkable predictive performance across various domains. However, these models frequently suffer from poor uncertainty calibration, producing overconfident predictions, particularly when confronted with o...
Elena Khan· International Journal of Com...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.