Superposition unifies power-law training dynamics

• URL: http://arxiv.org/abs/2602.01045v1
• Date: Sun, 01 Feb 2026 06:05:18 GMT
• Title: Superposition unifies power-law training dynamics
• Abstract summary: We investigate the role of feature superposition in the emergence of power-law training dynamics.<n>We first derive an analytic theory for training without superposition.<n>We discover that a superposition bottleneck induces a transition to a universal power-law exponent of $sim 1$.
• Score: 6.874132000933929
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models.

Related papers

• SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws [55.81298869922287]
  We study the training dynamics of multiclass logistic regression on high-dimensional Gaussian mixture models with a large number of classes.<n>We establish precise scaling laws governing the cross-entropy risk under gradient-based optimization.
  arXiv  Detail & Related papers  (2026-09-07T18:30:09Z)
• Interpreting learning dynamics of autoencoders: Transient scaling and emerging concepts of the Ising model [0.0]
  unsupervised autoencoders trained on microscopic spin configurations from the Ising model learn macroscopic, theory-relevant variables.<n>We quantify learning across multiple spatial scales and reveal two distinct dynamical regimes.<n>A dynamical viewpoint of learning is established in which intrinsic properties expose the effects of forced changes in representation.
  arXiv  Detail & Related papers  (2026-07-11T12:31:34Z)
• The Power of Power Law: Asymmetry Enables Compositional Reasoning [76.90253944094773]
  We show that learning under a power-law distribution provably requires significantly less training data.<n>Our results offer an alternative perspective on what constitutes an effective data distribution for training models.
  arXiv  Detail & Related papers  (2026-04-24T18:49:08Z)
• LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning [54.6434454250226]
  LongAct is a strategy that shifts from uniform to saliency-guided sparse updates.<n>By selectively updating only the weights associated with these significant activations, LongAct achieves an approximate 8% improvement on LongBench v2.
  arXiv  Detail & Related papers  (2026-04-16T12:06:59Z)
• A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning [51.505728136705564]
  We develop an analytical theory of the pretraining-fine-tuning pipeline in diagonal linear networks.<n>We find that different initialization choices place the network into four distinct fine-tuning regimes.<n>A smaller scale in earlier layers enables the network to both reuse and refine its features, leading to superior generalization.
  arXiv  Detail & Related papers  (2026-02-23T17:19:33Z)
• Fast State-Augmented Learning for Wireless Resource Allocation with Dual Variable Regression [83.27791109672927]
  We show how a state-augmented graph neural network (GNN) parametrization for the resource allocation policy circumvents the drawbacks of the ubiquitous dual subgradient methods.<n>Lagrangian maximizing state-augmented policies are learned during the offline training phase.<n>We prove a convergence result and an exponential probability bound on the excursions of the dual function (iterate) optimality gaps.
  arXiv  Detail & Related papers  (2025-06-23T15:20:58Z)
• Models of Heavy-Tailed Mechanistic Universality [62.107333654304014]
  We propose a family of random matrix models to explore attributes that give rise to heavy-tailed behavior in trained neural networks.<n>Under this model, spectral densities with power laws on tails arise through a combination of three independent factors.<n> Implications of our model on other appearances of heavy tails, including neural scaling laws, trajectories, and the five-plus-one phases of neural network training, are discussed.
  arXiv  Detail & Related papers  (2025-06-04T00:55:01Z)
• The emergence of sparse attention: impact of data distribution and benefits of repetition [14.652502263025882]
  We study the emergence over training of sparse attention, a critical and frequently observed attention pattern in Transformers.<n>By combining theoretical analysis of a toy model with empirical observations on small Transformers trained on a linear regression variant, we uncover the mechanics sparse attention emergence.<n>Our findings provide a simple, theoretically grounded framework for understanding how data distributions and model design influence the learning dynamics behind one form of emergence.
  arXiv  Detail & Related papers  (2025-05-23T13:14:02Z)
• Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks [12.108234998867337]
  We identify two novel textitdynamical scaling laws that govern how performance evolves as function of different norm-based complexity measures.<n>Our findings are consistent across CNNs, ResNets, and Vision Transformers trained on MNIST, CIFAR-10 and CIFAR-100.<n>We provide analytical support using a single-layer perceptron trained with logistic loss, where we derive the new dynamical scaling laws, and we explain them through the implicit bias induced by gradient-based training.
  arXiv  Detail & Related papers  (2025-05-19T15:13:36Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data.<n>Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra [0.0]
  Neural scaling laws describe how the performance of deep neural networks scales with key factors such as training data size, model complexity, and training time. We employ techniques from statistical mechanics to analyze one-pass gradient descent within a student-teacher framework.
  arXiv  Detail & Related papers  (2024-10-11T17:21:42Z)
• SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representations [76.45009891152178]
  Pretraining-finetuning approach can alleviate the labeling burden by fine-tuning a pre-trained backbone across various downstream datasets as well as tasks.<n>We show, for the first time, that general representations learning can be achieved through the task of occupancy prediction.<n>Our findings will facilitate the understanding of LiDAR points and pave the way for future advancements in LiDAR pre-training.
  arXiv  Detail & Related papers  (2023-09-19T11:13:01Z)
• The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles [24.52890377175555]
  We propose a general-purpose training strategy for transformers that can reduce both the memory and computational cost of self-attention by 4 to 8 times during training. We show that an ensemble of sub-models can be formed from the subsampled pathways within a network, which can achieve better performance than its densely attended counterpart.
  arXiv  Detail & Related papers  (2023-06-02T17:28:46Z)
• What training reveals about neural network complexity [80.87515604428346]
  This work explores the hypothesis that the complexity of the function a deep neural network (NN) is learning can be deduced by how fast its weights change during training. Our results support the hypothesis that good training behavior can be a useful bias towards good generalization.
  arXiv  Detail & Related papers  (2021-06-08T08:58:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.