Jan 2026· Physical Review E· Vol 114 2-2, pp.
025301
· 1 citation· 61 references
MedicinePhysicsComputer Science
TL;DR
This work explores the problem of prediction in the extrapolation regime of scale-free processes, and considers two paradigmatic problems that are statistically self-similar: the Abelian sandpile model and a two-dimensional fractional Gaussian field obeying linear dynamics.
Abstract
Machine learning has deeply changed some fields recently, like language and vision. In the case of complex systems, spectacular breakthroughs happened too (e.g., for protein folding) and more are expected to come. Our question is: how and to which extent can one regress scale-free processes, i.e., processes displaying power law behavior, like earthquakes or avalanches? The events one is interested in predicting are the large ones, i.e., events that are typically rare in the training set, so we are basically in the extrapolation regime. While some recent works also tackle scale-free systems by proposing generative models closely aligned with the renormalization group framework, here instead we explore the problem of prediction in the extrapolation regime. We consider two paradigmatic problems that are statistically self-similar. The first one is a two-dimensional fractional Gaussian field obeying linear dynamics, self-similar by construction and amenable to exact analysis. The second one is the Abelian sandpile model, exhibiting self-organized criticality. The emerging paradigm of geometric deep learning shows that including known symmetries into the model's architecture is key to success (as translation invariance for images). Here one may hope to extrapolate only by leveraging scale invariance, which is, however, a peculiar symmetry, as it involves possibly nontrivial coarse-graining operations and anomalous scaling. We perform experiments on various existing architectures like U-net, the Riesz network (scale-invariant by construction), or our own proposals: a wavelet-decomposition-based Graph Neural Network (with discrete scale symmetry), a Fourier embedding layer, and a Fourier-Mellin Neural Operator. Based on these experiments and a complete characterization of the linear case, we identify the main issues relative to spectral biases and coarse-grained representations, and discuss how to alleviate them with the relevant inductive biases.
Analytically, the spectrum is solved in the lazy high-dimensional limit for both linear ($n \asymp d) and polynomial ($n \asymp d^k$) sample complexities, and it is proved through a bias--variance decomposition that the first bulk minimizes the approximation error while the second drives the error associated with memor...
Raphaël Urfin, T. Bonnaire, G. Biroli et al.· arXiv.org· 1 citation
The ability to derive the governing equations of a dynamical system from data is essential for prediction and control across many scientific fields. Two challenges, however, are the expansion of candidate terms when reducing higher-order equations to first-order form and noise from estimating derivatives. We propose Hi...
Despite the widespread use of Low-Rank Adaptation (LoRA), little is known about its dynamics in continual learning and the mechanisms by which low-rank updates affect catastrophic forgetting. We provide an asymptotically exact dynamical characterization of LoRA in a solvable two-task teacher-student model. In the high-...
Théo Marchetta, F. Alessandroni, Alessandro Breccia et al.· 1 citation
To get good performance for two-dimensional site percolation developping a supervised, scale-shared neural architecture, it is key that the learned latent representation exhibits critical fluctuations and scale-dependent flows consistent with the renormalization-group structure of percolation.
Anaclara Alvez, L. Camagna, S. Chibbaro et al.· 1 citation
Theoretical scaling laws from linear regression to multiclass classification to multiclass classification are extended, while connecting to empirical scaling laws observed in large-scale neural networks.
Konstantinos Christopher Tsiolis, Denny Wu, Christos Thrampoulidis et al.· 0 citations
It is shown that a symmetry fixes what variables are: a network layer is a sum over interchangeable units, so relabeling the units leaves it unchanged; given smoothness and the condition that a unit's gradient vanish at the origin, symmetry then enforces a universal leading form for the expansion about the near-zero we...
Zi-Yin Liu, Yi-Zhou Xu, Tomaso A. Poggio et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026