Learning and extrapolating scale-invariant processes

Abstract

Machine learning has deeply changed some fields recently, like language and vision. In the case of complex systems, spectacular breakthroughs happened too (e.g., for protein folding) and more are expected to come. Our question is: how and to which extent can one regress scale-free processes, i.e., processes displaying power law behavior, like earthquakes or avalanches? The events one is interested in predicting are the large ones, i.e., events that are typically rare in the training set, so we are basically in the extrapolation regime. While some recent works also tackle scale-free systems by proposing generative models closely aligned with the renormalization group framework, here instead we explore the problem of prediction in the extrapolation regime. We consider two paradigmatic problems that are statistically self-similar. The first one is a two-dimensional fractional Gaussian field obeying linear dynamics, self-similar by construction and amenable to exact analysis. The second one is the Abelian sandpile model, exhibiting self-organized criticality. The emerging paradigm of geometric deep learning shows that including known symmetries into the model's architecture is key to success (as translation invariance for images). Here one may hope to extrapolate only by leveraging scale invariance, which is, however, a peculiar symmetry, as it involves possibly nontrivial coarse-graining operations and anomalous scaling. We perform experiments on various existing architectures like U-net, the Riesz network (scale-invariant by construction), or our own proposals: a wavelet-decomposition-based Graph Neural Network (with discrete scale symmetry), a Fourier embedding layer, and a Fourier-Mellin Neural Operator. Based on these experiments and a complete characterization of the linear case, we identify the main issues relative to spectral biases and coarse-grained representations, and discuss how to alleviate them with the relevant inductive biases.