The Memory-Efficient Neural Operator (MENO) is proposed as a high-performance PDE neural solver based on the Manifold Function Encoder (MFE), which has a significantly smaller memory footprint and much faster training speed than other popular architectures.
Abstract
We propose the Memory-Efficient Neural Operator (MENO) as a high-performance PDE neural solver based on the Manifold Function Encoder (MFE). MENO features three primary advantages: (1) MENO has a significantly smaller memory footprint and much faster training speed than other popular architectures, with the memory footprint being independent of the data resolution, and therefore holds the potential for scaling up to large-scale models. (2) MENO can accept PDE inputs of arbitrary form, including arbitrary geometric domains and arbitrary discretizations. In particular, it is capable of handling cross-geometry scenarios, i.e., where the input functions and the output solutions are defined on different manifolds. (3) MENO exhibits strong generalization capability, and achieves the best accuracy on most of the benchmarks we tested, compared with the results reported in the literature. The code is available on GitHub at https://github.com/jpzxshi/MENO, and all numerical examples in this paper can be run with a single command to reproduce the reported results.
GPU memory is increasingly the primary bottleneck in scaling deep neural network (DNN) training, where the activation tensors footprint of a model may exceed the memory capacity. Tensor recomputation is a powerful technique that trades additional computation for reduced peak memory usage. However, existing approaches f...
Dat Nguyen, V. Devarakonda, An-Xiao Jiang et al.· Proceedings of the ACM on Pr...· 0 citations
The Levenberg-Marquardt (LMA) algorithm is a widely used nonlinear least-squares solver known for its rapid convergence. For small-scale neural networks, LMA provides better convergence and architectural robustness than first-order optimizers like Adam. However, LMA imposes prohibitive memory and computational costs. T...
Xiang-Hao Wang, Johan Barthélemy, John A. Taylor· 0 citations
Training neural networks in reduced precision can yield significant efficiency gains, yet all existing methods that scale to standard computer vision benchmarks retain floating-point in at least one stage, typically the weight update or optimizer state. We present a training algorithm that scales to modern deep neural...
Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel Roveri· 0 citations
This work proposes LocalProp, a training procedure that locally updates the weights of a model, where the pre-training is based on I-JEPA, and empirically shows that it reaches good performance at a fraction of GPU peak memory.
Diana-Nicoleta Grigore, Iuliana Georgescu, R. Ionescu· arXiv.org· 0 citations
The Interpolating Neural Operator (INO) is proposed, a data-free interpolating neural network that is trained directly on the weak form of the PDE, and is compared with physics-informed FNO and DeepONet on different benchmarks.
Jia-Chen Guo, Ye Lu, Nai-Chen Shi et al.· arXiv.org· 0 citations
Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus efficient on standard hardware. Existing methods, however, choose the subspace to remove from each weight matrix with local closed-form crit...
Massimo Bini, Anders Christensen, Stephan Alaniz et al.· arXiv.org· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026