The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks

• URL: http://arxiv.org/abs/2605.06258v1
• Date: Thu, 07 May 2026 13:35:11 GMT
• Title: The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks
• Abstract summary: Understanding how deep neural networks learn remains a central challenge in machine learning theory.<n>We propose a feature-centric framework for analyzing neural network training by relating weight updates to feature evolution.<n>By analyzing the training and layer-wise dynamics, we show that deep networks learn to sequentially transform representations toward target-linear structure.
• Score: 10.475180266869081
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Understanding how deep neural networks learn representations remains a central challenge in machine learning theory. In this work, we propose a feature-centric framework for analyzing neural network training by relating weight updates to feature evolution. We introduce a simple identity, the Feature Learning Equation, which identifies the weight Gram matrix as the key object capturing feature dynamics. This enables us to interpret gradient descent as implicitly inducing a hypothetical evolution of features, whose covariance structure - termed the Virtual Covariance - characterizes how representations evolve during training. Building on this perspective, we introduce Target Linearity, a measure quantifying the linear alignment between features and targets. By analyzing the training and layer-wise dynamics, we show that deep networks learn to sequentially transform representations toward target-linear structure. This linearization perspective provides a unified interpretation of several empirical phenomena, including Neural Collapse and linear interpolation in generative models.

Related papers

• Spectral phase transitions and trainability in neural network learning dynamics [0.0]
  We study the emergence of low-dimensional structures in the spectra of neural network weight matrices during training.<n>Our results suggest that spectral analysis may provide a unified perspective of learning dynamics.
  arXiv  Detail & Related papers  (2026-06-26T18:00:01Z)
• Concept-Guided Interpretability via Neural Chunking [64.6429903327095]
  We show that neural networks exhibit patterns in their raw population activity that mirror regularities in the training data.<n>We propose three methods to extract recurring chunks on a neural population level.<n>Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data.
  arXiv  Detail & Related papers  (2025-05-16T13:49:43Z)
• An Analytical Characterization of Sloppiness in Neural Networks: Insights from Linear Models [18.99511760351873]
  Recent experiments have shown that training trajectories of multiple deep neural networks evolve on a remarkably low-dimensional "hyper-ribbon-like" manifold.<n>Inspired by the similarities in the training trajectories of deep networks and linear networks, we analytically characterize this phenomenon for the latter.<n>We show that the geometry of this low-dimensional manifold is controlled by (i) the decay rate of the eigenvalues of the input correlation matrix of the training data, (ii) the relative scale of the ground-truth output to the weights at the beginning of training, and (iii) the number of steps of gradient descent.
  arXiv  Detail & Related papers  (2025-05-13T19:20:19Z)
• Predicting Steady-State Behavior in Complex Networks with Graph Neural Networks [0.0]
  In complex systems, information propagation can be defined as diffused or delocalized, weakly localized, and strongly localized.<n>This study investigates the application of graph neural network models to learn the behavior of a linear dynamical system on networks.
  arXiv  Detail & Related papers  (2025-02-02T17:29:10Z)
• Network Dynamics-Based Framework for Understanding Deep Neural Networks [8.376259475051974]
  We propose a theoretical framework to analyze learning dynamics through the lens of dynamical systems theory.<n>We redefine the notions of linearity and nonlinearity in neural networks by introducing two fundamental transformation units at the neuron level.<n>Different transformation modes lead to distinct collective behaviors in weight vector organization, different modes of information extraction, and the emergence of qualitatively different learning phases.
  arXiv  Detail & Related papers  (2025-01-05T04:23:21Z)
• Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
  In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
  arXiv  Detail & Related papers  (2024-10-31T22:54:34Z)
• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Dynamical stability and chaos in artificial neural network trajectories along training [3.379574469735166]
  We study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network. We find hints of regular and chaotic behavior depending on the learning rate regime. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning.
  arXiv  Detail & Related papers  (2024-04-08T17:33:11Z)
• Binding Dynamics in Rotating Features [72.80071820194273]
  We propose an alternative "cosine binding" mechanism, which explicitly computes the alignment between features and adjusts weights accordingly. This allows us to draw direct connections to self-attention and biological neural processes, and to shed light on the fundamental dynamics for object-centric representations to emerge in Rotating Features.
  arXiv  Detail & Related papers  (2024-02-08T12:31:08Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.