Skip to content

Fisher8: Stabilizing Neural Heteroscedastic Regression via Output-Layer Fisher Geometry

Aug 2026 · arXiv.org · Vol abs/2608.10374 · 0 citations · 22 references
Computer Science

TL;DR

Fisher8 is derived, an output-layer gradient correction that reorients and rescales updates using Fisher geometry rather than Euclidean geometry, and it is shown that prior stabilizers converge on overlapping components of this geometric correction.

Abstract

Training neural networks to jointly predict mean and uncertainty estimates from noisy observations can be unstable, prompting a series of independent stabilization efforts. We argue that these interventions highlight a common underlying issue where gradient steps are poorly aligned with the geometry of the loss landscape. To better align updates with local curvature, we derive Fisher8, an output-layer gradient correction that reorients and rescales updates using Fisher geometry rather than Euclidean geometry. Unlike past stabilizers, Fisher8 introduces no data-dependent hyperparameters beyond learning rate and admits an approximate KL trust radius between successive predictive distributions. We show that prior stabilizers converge on overlapping components of this geometric correction. Across multidimensional regression and representation-learning tasks, Fisher8 obtains superior likelihood--error tradeoffs, predicts calibrated uncertainty estimates, and learns rich uncertainty-aware feature spaces.

View source

Similar papers

Preprint Oct 2026

Does LLM Pre-Training Typically Occur at the Edge of Stability?

Quadratic approximations provide a common lens for studying neural network optimization, but recent evidence challenges their predictive validity. In full-batch gradient descent with learning rate (LR) $\eta$, Cohen et al. (2021) observed the Edge of Stability (EoS), where the largest Hessian eigenvalue concentrates ne...

Yu-Hang Cai, Hao-Dong Wen, Hao-Feng Huang et al. · 2 citations

Understanding LLM Parameter Update Sparsity through the Lens of Fisher

Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training...

Yu-Fan Zhang, Sagnik Mukherjee, Hao Peng · 1 citation

Optimal Transport Dropout for Structured Predictive Uncertainty

Optimal Transport Dropout (OTD) is introduced, which offers a way to learn structured predictive uncertainty without explicit posterior inference or ensembles of independently trained predictors.

Giacomo Lorenzon, Francesco Regazzoni · 0 citations
#machine learning Preprint Oct 2026

Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization

Recent successes of matrix- and curvature-based optimizers have renewed interest in how update geometry shapes learning. These methods normalize or precondition updates, changing how different components progress during training. In deep matrix factorization, the geometry that slows gradient descent (GD) favors low-ran...

Mahalakshmi Sabanayagam, S. Lucey · 1 citation
#machine learning Preprint Oct 2026

Learn Only Where You Can't Compute: Structured Corrections to Gaussian Process Posteriors

Physics-informed Gaussian processes (GPs) provide a principled framework for scientific inference by expressing uncertainty over empirical measurements, boundary conditions, and governing equations. However, they struggle to scale to larger datasets and nonlinear information. Conversely, diffusion models excel at learn...

Tobias Weber, Carla Sagebiel, Philipp Hennig · 0 citations
Open access Sep 2026

Probabilistic Shrinkage Priors for Calibrated Representation Spaces

Modern deep learning architectures excel at learning complex representations from high-dimensional data, achieving remarkable predictive performance across various domains. However, these models frequently suffer from poor uncertainty calibration, producing overconfident predictions, particularly when confronted with o...

Elena Khan · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.