Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks

• URL: http://arxiv.org/abs/2608.06597v1
• Date: Thu, 06 Aug 2026 21:15:32 GMT
• Title: Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
• Abstract summary: A scientific theory of deep learning, comprising learning dynamics and statistical properties of learned models, is rapidly gaining attention.<n>One of the corner stones of this development are analytically solvable toy models, allowing for the fully tractable analysis of the learning dynamics.<n>Here we analytically investigate such a toy model using the regularization strength as a tunable external parameter - akin to external fields in statistical physics.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A scientific theory of deep learning, comprising learning dynamics and statistical properties of learned models, is rapidly gaining attention. One of the corner stones of this development are analytically solvable toy models, allowing for the fully tractable analysis of the learning dynamics. Here we analytically investigate such a toy model using the regularization strength as a tunable external parameter - akin to external fields in statistical physics. In previous studies, (i) an onset of learning transition was predicted analytically and (ii) it was phenomenologically/numerically established that tuning the regularization strength can result in a cascade of phase transitions. The number of those transitions was linked to the geometry of the loss landscape determined by the model complexity. Setting up a rigorous framework underpinning the previous numerical observations, our investigation reveals a precise connection between those cascades of phase transitions, learnable features and the underlying geometry. We provide analytic predictions of these phase transitions as well as tractable order parameters related to learned features. At the level of the minimal model, we connect this macroscopic perspective (that can be condensed into an effective description) to the microscopic perspective in terms of the geometry of the loss landscape characterized by the Hessian spectrum. Thus, the presented model provides a platform to explore and sharpen advances made in the scientific theory of deep learning rooted in statistical physics concepts.

Related papers

• Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks [50.79340930913805]
  We study a class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning.<n>By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.
  arXiv  Detail & Related papers  (2026-07-13T17:56:11Z)
• Causal Representation Meets Stochastic Modeling under Generic Geometry [49.24293444627916]
  We develop causal representation learning for continuous-time latent point processes.<n>We develop MUTATE, an identifiable variational autoencoder framework with a time-adaptive transition module.<n>Across simulated and empirical studies, we find that MUTATE can effectively answer scientific questions.
  arXiv  Detail & Related papers  (2026-02-04T20:40:53Z)
• Phase transitions reveal hierarchical structure in deep neural networks [0.0]
  We show that phase transitions in Deep Neural Networks are governed by saddle points in the loss landscape.<n>We introduce a simple, fast, and easy to implement algorithm that uses the L2 regularizer as a tool to probe the geometry of error landscapes.
  arXiv  Detail & Related papers  (2025-12-05T15:14:09Z)
• Neural models for prediction of spatially patterned phase transitions: methods and challenges [0.37282630026096597]
  Early Warning Signal (EWS) detection has shown promise in identifying dynamical signatures of oncoming critical transitions.<n>This paper explores the successes and shortcomings of neural EWS detection for spatially phase patterned transitions.
  arXiv  Detail & Related papers  (2025-05-14T18:24:15Z)
• Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
  In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
  arXiv  Detail & Related papers  (2024-10-31T22:54:34Z)
• Analyzing Deep Transformer Models for Time Series Forecasting via Manifold Learning [4.910937238451485]
  Transformer models have consistently achieved remarkable results in various domains such as natural language processing and computer vision. Despite ongoing research efforts to better understand these models, the field still lacks a comprehensive understanding. Time series data, unlike image and text information, can be more challenging to interpret and analyze.
  arXiv  Detail & Related papers  (2024-10-17T17:32:35Z)
• Cascade of phase transitions in the training of Energy-based models [9.945465034701288]
  We investigate the feature encoding process in a prototypical energy-based generative model, the Bernoulli-Bernoulli RBM.<n>Our study tracks the evolution of the model's weight matrix through its singular value decomposition.<n>We validate our theoretical results by training the Bernoulli-Bernoulli RBM on real data sets.
  arXiv  Detail & Related papers  (2024-05-23T15:25:56Z)
• Explaining the Machine Learning Solution of the Ising Model [0.0]
  This work shows how it can be accomplished for the ferromagnetic Ising model, the main target of several machine learning (ML) studies in statistical physics. By using a neural network (NN) without hidden layers (the simplest possible) and informed by the symmetry of the Hamiltonian, an explanation is provided for the strategy used in finding the supervised learning solution. These results pave the way to a physics-informed explainable generalized framework, enabling the extraction of physical laws and principles from the parameters of the models.
  arXiv  Detail & Related papers  (2024-02-18T20:47:33Z)
• Neural network analysis of neutron and X-ray reflectivity data: Incorporating prior knowledge for tackling the phase problem [141.5628276096321]
  We present an approach that utilizes prior knowledge to regularize the training process over larger parameter spaces. We demonstrate the effectiveness of our method in various scenarios, including multilayer structures with box model parameterization. In contrast to previous methods, our approach scales favorably when increasing the complexity of the inverse problem.
  arXiv  Detail & Related papers  (2023-06-28T11:15:53Z)
• Latent Traversals in Generative Models as Potential Flows [113.4232528843775]
  We propose to model latent structures with a learned dynamic potential landscape. Inspired by physics, optimal transport, and neuroscience, these potential landscapes are learned as physically realistic partial differential equations. Our method achieves both more qualitatively and quantitatively disentangled trajectories than state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-04-25T15:53:45Z)
• Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
  We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
  arXiv  Detail & Related papers  (2020-02-20T15:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.