Skip to content

Linearized subspace refinement framework to expose hidden accuracy in trained neural networks

Jan 2026 · 0 citations
Computer Science

TL;DR

This work presents Linearized Subspace Refinement (LSR), an architecture-agnostic post-training framework that exploits the local linearized model at a fixed trained state and exposes conditioning-limited attainable accuracy in trained-state linearized models and provides direct access to it.

Abstract

Neural networks trained by gradient-based methods often exhibit optimization-induced accuracy plateaus in scientific machine learning tasks. We present Linearized Subspace Refinement (LSR), an architecture-agnostic post-training framework that exploits the local linearized model at a fixed trained state. By solving a reduced direct least-squares problem in a Jacobian-defined low-dimensional space, LSR computes a subspace-optimal linearized correction and yields a refined predictor with markedly improved accuracy. Across function approximation, data-driven operator learning, physics-informed operator fine-tuning, and noisy inverse problems, LSR shows that standard nonlinear training can remain far above this subspace-attainable error level. Similar accuracy plateaus persist even for the convex quadratic problem from local linearization when solved with standard iterative optimizers, identifying numerical ill-conditioning as a primary bottleneck. LSR frequently delivers order-of-magnitude error reductions, while the subspace rank provides an explicit capacity-control mechanism that balances correction strength, numerical stability, and noise sensitivity. Together, LSR exposes conditioning-limited attainable accuracy in trained-state linearized models and provides direct access to it.

View source

Similar papers

Gradient-estimator design overcomes trainability barriers in neural-network-based variational optimization

An unbiased direct gradient estimator is derived and the adaptive minimum-variance phase (AMVP) estimator is introduced for neural-network variational optimization by improving the signal-to-noise ratio of weak gradients to enable reliable scientific calculations where training previously failed, while substantially re...

Yi-He Xue, Rui Wang, Bai-Geng Wang et al. · 0 citations
#machine learning Preprint Sep 2026

Local gradient neural operator

This work proposes the local gradient neural operator (LGNO) as a lightweight and interpretable alternative for field temporal evolution prediction and source identification in typical mechanical problems and exhibits wide applicability to mechanical problems including diffusion, flow, and quantum phenomena.

Bai-Ming Zhang, Jin-Song Tang, Ying Xu et al. · 0 citations
Open access Aug 2026

Provably convergent proximal gradient methods with tailored input convex neural networks regularization for inverse problems

This study presents a novel Deep Proximal Gradient Descent framework for ill-posed problems by employing a tailored second-order differentiable Input-Convex Neural Networks (ICNNs) as a learned regularizer, and introduces an innovative formulation that employs the learned residual to guide gradient descent.

Tian-Yi Ye, Guang-Yu Gao, Yang Li et al. · 0 citations
Aug 2026

Fisher8: Stabilizing Neural Heteroscedastic Regression via Output-Layer Fisher Geometry

Fisher8 is derived, an output-layer gradient correction that reorients and rescales updates using Fisher geometry rather than Euclidean geometry, and it is shown that prior stabilizers converge on overlapping components of this geometric correction.

Sumedh Vemuganti, Nickvash Kani · 0 citations

SINO: Scale-Invariant Neural Operator

In scientific machine learning, physical fields governed by partial differential equations exhibit low-rank structure and scale invariance. When solving equations on coarse grids, missing information leads to the closure problem: modeling unresolved physics to recover lost dynamics. Although closure terms depend on gri...

Kai-Chen Ouyang, Cheng-Lei Yu, Chuan-Rui Wang et al. · 0 citations
#machine learning Preprint Oct 2026

Operator-informed initialization for Fourier features physics-informed neural networks

Physics-Informed Neural Networks (PINNs) typically exhibit spectral bias, where some frequencies of the target function converge more slowly than others. In this work, we analyze the training dynamics of Fourier Feature PINNs in the Neural Tangent Kernel regime to address this limitation. We derive an explicit evolutio...

Juan Molina, P. Perdikaris, Mircea Petrache et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.