This work presents Linearized Subspace Refinement (LSR), an architecture-agnostic post-training framework that exploits the local linearized model at a fixed trained state and exposes conditioning-limited attainable accuracy in trained-state linearized models and provides direct access to it.
Abstract
Neural networks trained by gradient-based methods often exhibit optimization-induced accuracy plateaus in scientific machine learning tasks. We present Linearized Subspace Refinement (LSR), an architecture-agnostic post-training framework that exploits the local linearized model at a fixed trained state. By solving a reduced direct least-squares problem in a Jacobian-defined low-dimensional space, LSR computes a subspace-optimal linearized correction and yields a refined predictor with markedly improved accuracy. Across function approximation, data-driven operator learning, physics-informed operator fine-tuning, and noisy inverse problems, LSR shows that standard nonlinear training can remain far above this subspace-attainable error level. Similar accuracy plateaus persist even for the convex quadratic problem from local linearization when solved with standard iterative optimizers, identifying numerical ill-conditioning as a primary bottleneck. LSR frequently delivers order-of-magnitude error reductions, while the subspace rank provides an explicit capacity-control mechanism that balances correction strength, numerical stability, and noise sensitivity. Together, LSR exposes conditioning-limited attainable accuracy in trained-state linearized models and provides direct access to it.
An unbiased direct gradient estimator is derived and the adaptive minimum-variance phase (AMVP) estimator is introduced for neural-network variational optimization by improving the signal-to-noise ratio of weak gradients to enable reliable scientific calculations where training previously failed, while substantially re...
Yi-He Xue, Rui Wang, Bai-Geng Wang et al.· arXiv.org· 0 citations
This work proposes the local gradient neural operator (LGNO) as a lightweight and interpretable alternative for field temporal evolution prediction and source identification in typical mechanical problems and exhibits wide applicability to mechanical problems including diffusion, flow, and quantum phenomena.
Bai-Ming Zhang, Jin-Song Tang, Ying Xu et al.· 0 citations
This study presents a novel Deep Proximal Gradient Descent framework for ill-posed problems by employing a tailored second-order differentiable Input-Convex Neural Networks (ICNNs) as a learned regularizer, and introduces an innovative formulation that employs the learned residual to guide gradient descent.
Tian-Yi Ye, Guang-Yu Gao, Yang Li et al.· Inverse Problems· 0 citations
Fisher8 is derived, an output-layer gradient correction that reorients and rescales updates using Fisher geometry rather than Euclidean geometry, and it is shown that prior stabilizers converge on overlapping components of this geometric correction.
In scientific machine learning, physical fields governed by partial differential equations exhibit low-rank structure and scale invariance. When solving equations on coarse grids, missing information leads to the closure problem: modeling unresolved physics to recover lost dynamics. Although closure terms depend on gri...
Kai-Chen Ouyang, Cheng-Lei Yu, Chuan-Rui Wang et al.· arXiv.org· 0 citations
Physics-Informed Neural Networks (PINNs) typically exhibit spectral bias, where some frequencies of the target function converge more slowly than others. In this work, we analyze the training dynamics of Fourier Feature PINNs in the Neural Tangent Kernel regime to address this limitation. We derive an explicit evolutio...
Juan Molina, P. Perdikaris, Mircea Petrache et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026