Learning and extrapolating scale-invariant processes

• URL: http://arxiv.org/abs/2601.14810v1
• Date: Wed, 21 Jan 2026 09:35:44 GMT
• Title: Learning and extrapolating scale-invariant processes
• Authors: Anaclara Alvez-Canepa, Cyril Furtlehner, François Landes,
• Abstract summary: We tackle the question of how and to which extent can one regress scale-free processes, i.e. processes displaying power law behavior, like earthquakes or avalanches?<n>We are interested in predicting the large ones, i.e. rare events in the training set which therefore require extrapolation capabilities of the model.
• Score: 3.331543293568139
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Machine Learning (ML) has deeply changed some fields recently, like Language and Vision and we may expect it to be relevant also to the analysis of of complex systems. Here we want to tackle the question of how and to which extent can one regress scale-free processes, i.e. processes displaying power law behavior, like earthquakes or avalanches? We are interested in predicting the large ones, i.e. rare events in the training set which therefore require extrapolation capabilities of the model. For this we consider two paradigmatic problems that are statistically self-similar. The first one is a 2-dimensional fractional Gaussian field obeying linear dynamics, self-similar by construction and amenable to exact analysis. The second one is the Abelian sandpile model, exhibiting self-organized criticality. The emerging paradigm of Geometric Deep Learning shows that including known symmetries into the model's architecture is key to success. Here one may hope to extrapolate only by leveraging scale invariance. This is however a peculiar symmetry, as it involves possibly non-trivial coarse-graining operations and anomalous scaling. We perform experiments on various existing architectures like U-net, Riesz network (scale invariant by construction), or our own proposals: a wavelet-decomposition based Graph Neural Network (with discrete scale symmetry), a Fourier embedding layer and a Fourier-Mellin Neural Operator. Based on these experiments and a complete characterization of the linear case, we identify the main issues relative to spectral biases and coarse-grained representations, and discuss how to alleviate them with the relevant inductive biases.

Related papers

• Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning [52.26396748560348]
  We provide an overview of high dimensional dynamical systems driven by random matrices.<n>We focus on applications to simple models of learning and generalization in machine learning theory.
  arXiv  Detail & Related papers  (2026-01-03T00:12:32Z)
• Scaling Laws and Symmetry, Evidence from Neural Force Fields [14.109815254143205]
  We show a clear power-law scaling behaviour with respect to data, parameters and compute with architecture-dependent exponents''<n>In particular, we observe that equivariant architectures, which leverage task symmetry, scale better than non-equivariant models.<n>Our analysis also suggests that for compute-optimal training, the data and model sizes should scale in tandem regardless of the architecture.
  arXiv  Detail & Related papers  (2025-10-10T18:22:00Z)
• Bayesian Double Descent [0.0]
  We show that deep neural networks have a re-descending property in their risk function.<n>As the complexity of the model increases, risk exhibits a U-shaped region.<n>As the number of parameters equals the number of observations and the model becomes one of where the risk can be unbounded, it re-descends.
  arXiv  Detail & Related papers  (2025-07-09T23:47:26Z)
• Generalized Linear Mode Connectivity for Transformers [87.32299363530996]
  A striking phenomenon is linear mode connectivity (LMC), where independently trained models can be connected by low- or zero-loss paths.<n>Prior work has predominantly focused on neuron re-ordering through permutations, but such approaches are limited in scope.<n>We introduce a unified framework that captures four symmetry classes: permutations, semi-permutations, transformations, and general invertible maps.<n>This generalization enables, for the first time, the discovery of low- and zero-barrier linear paths between independently trained Vision Transformers and GPT-2 models.
  arXiv  Detail & Related papers  (2025-06-28T01:46:36Z)
• Analyzing Deep Transformer Models for Time Series Forecasting via Manifold Learning [4.910937238451485]
  Transformer models have consistently achieved remarkable results in various domains such as natural language processing and computer vision. Despite ongoing research efforts to better understand these models, the field still lacks a comprehensive understanding. Time series data, unlike image and text information, can be more challenging to interpret and analyze.
  arXiv  Detail & Related papers  (2024-10-17T17:32:35Z)
• From system models to class models: An in-context learning paradigm [0.0]
  We introduce a novel paradigm for system identification, addressing two primary tasks: one-step-ahead prediction and multi-step simulation. We learn a meta model that represents a class of dynamical systems. For one-step prediction, a GPT-like decoder-only architecture is utilized, whereas the simulation problem employs an encoder-decoder structure.
  arXiv  Detail & Related papers  (2023-08-25T13:50:17Z)
• Geometric Neural Diffusion Processes [55.891428654434634]
  We extend the framework of diffusion models to incorporate a series of geometric priors in infinite-dimension modelling. We show that with these conditions, the generative functional model admits the same symmetry.
  arXiv  Detail & Related papers  (2023-07-11T16:51:38Z)
• Learning Physical Dynamics with Subequivariant Graph Neural Networks [99.41677381754678]
  Graph Neural Networks (GNNs) have become a prevailing tool for learning physical dynamics. Physical laws abide by symmetry, which is a vital inductive bias accounting for model generalization. Our model achieves on average over 3% enhancement in contact prediction accuracy across 8 scenarios on Physion and 2X lower rollout MSE on RigidFall.
  arXiv  Detail & Related papers  (2022-10-13T10:00:30Z)
• Git Re-Basin: Merging Models modulo Permutation Symmetries [3.5450828190071655]
  We show how simple algorithms can be used to fit large networks in practice. We demonstrate the first (to our knowledge) demonstration of zero mode connectivity between independently trained models. We also discuss shortcomings in the linear mode connectivity hypothesis.
  arXiv  Detail & Related papers  (2022-09-11T10:44:27Z)
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training [4.318555434063275]
  Deep learning algorithms are responsible for a technological revolution in a variety of tasks including image recognition or Go playing. Yet, why they work is not understood. Ultimately, they manage to classify data lying in high dimension -- a feat generically impossible. We argue that different learning regimes can be organized into a phase diagram.
  arXiv  Detail & Related papers  (2020-12-30T11:00:36Z)
• Semiparametric Nonlinear Bipartite Graph Representation Learning with Provable Guarantees [106.91654068632882]
  We consider the bipartite graph and formalize its representation learning problem as a statistical estimation problem of parameters in a semiparametric exponential family distribution. We show that the proposed objective is strongly convex in a neighborhood around the ground truth, so that a gradient descent-based method achieves linear convergence rate. Our estimator is robust to any model misspecification within the exponential family, which is validated in extensive experiments.
  arXiv  Detail & Related papers  (2020-03-02T16:40:36Z)
• Learning Bijective Feature Maps for Linear ICA [73.85904548374575]
  We show that existing probabilistic deep generative models (DGMs) which are tailor-made for image data, underperform on non-linear ICA tasks. To address this, we propose a DGM which combines bijective feature maps with a linear ICA model to learn interpretable latent structures for high-dimensional data. We create models that converge quickly, are easy to train, and achieve better unsupervised latent factor discovery than flow-based models, linear ICA, and Variational Autoencoders on images.
  arXiv  Detail & Related papers  (2020-02-18T17:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.