Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts

• URL: http://arxiv.org/abs/2603.07323v1
• Date: Sat, 07 Mar 2026 20:07:16 GMT
• Title: Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts
• Abstract summary: We introduce the Norm-Hierarchy Transition (NHT) framework, which explains delayed representation learning.<n>We derive a tight bound showing that the transition delay grows logarithmically with the ratio between shortcut and structured norms.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks often rely on spurious shortcuts for many epochs before discovering structured representations. However, the mechanism governing when this transition occurs and whether its timing can be predicted remains unclear. Prior work shows that gradient descent converges to low norm solutions and that neural networks exhibit simplicity bias, but neither explains the timescale of the transition from shortcut features to structured representations. We introduce the Norm-Hierarchy Transition (NHT) framework, which explains delayed representation learning as the slow traversal of a hierarchy of parameter norms during regularized optimization. When multiple interpolating solutions exist with different norms, weight decay gradually moves the model from high norm shortcut solutions toward lower norm structured representations. We derive a tight bound showing that the transition delay grows logarithmically with the ratio between shortcut and structured norms. Experiments on modular arithmetic, CIFAR-10 with spurious features, CelebA, and Waterbirds support the predictions of the framework. The results suggest that grokking, shortcut learning, and delayed feature discovery arise from a common mechanism based on norm hierarchy traversal during training.

Related papers

• Accelerating Hierarchical Sparse Predictive Coding with Hybrid Amortized Inference [0.0]
  We show that a shallow LISTA-style initializer plus short corrective recurrence improves over pure amortization while remaining much faster than long iterative inference.<n>Results show that a shallow LISTA-style initializer plus short corrective recurrence improves over pure amortization while remaining much faster than long iterative inference.
  arXiv  Detail & Related papers  (2026-06-26T07:38:55Z)
• Monitoring Neural Training with Topology: A Footprint-Predictable Collapse Index [51.56484100374058]
  We propose an online, topology-aware monitor for evolving neural representations that couples Modular Morse Homology Maintenance (MMHM) with a composite Collapse Index (CI)<n>Instead of rebuilding complexes each epoch, we apply sparse edits at a fixed scale and maintain a discrete Morse matching, yielding fast, incremental updates.
  arXiv  Detail & Related papers  (2026-04-28T19:21:21Z)
• A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization [0.0]
  Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood.<n>We present a controlled study that systematically disentangles factors on modular addition (mod 97)<n>Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between stability and regularization.
  arXiv  Detail & Related papers  (2026-03-26T04:16:01Z)
• A Representation-Consistent Gated Recurrent Framework for Robust Medical Time-Series Classification [0.0]
  We propose a representation-consistent gated recurrent framework (RC-GRF) that introduces a principled regularization strategy to enforce temporal consistency in hidden-state representations.<n>The proposed framework is model-agnostic and can be integrated into existing gated recurrent architectures without modifying their internal gating mechanisms.
  arXiv  Detail & Related papers  (2026-02-10T17:16:49Z)
• Hierarchical temporal receptive windows and zero-shot timescale generalization in biologically constrained scale-invariant deep networks [1.2031796234206136]
  We train biologically constrained deep networks based on scale-invariant hippocampal time cells.<n>We find that a hierarchy of TRWs emerged naturally across layers, despite the network having an identical spectrum of time constants within layers.<n>We then distilled these inductive priors into a biologically plausible recurrent architecture, SITH-RNN.
  arXiv  Detail & Related papers  (2026-01-06T00:36:45Z)
• Generalization in Representation Models via Random Matrix Theory: Application to Recurrent Networks [7.721672385781673]
  We first study the generalization error of models that use a fixed feature representation (frozen intermediate layers) followed by a trainable readout layer.<n>We apply Random Matrix Theory to derive a closed-form expression for the generalization error.<n>We then apply this analysis to recurrent representations and obtain concise formula that characterize their performance.
  arXiv  Detail & Related papers  (2025-11-04T09:30:31Z)
• IBN: An Interpretable Bidirectional-Modeling Network for Multivariate Time Series Forecasting with Variable Missing [6.481926629151858]
  We propose the Interpretable Bidirectional-modeling Network (IBN)<n>IBN integrates Uncertainty-Aware Interpolation (UAI) and Graph Convolution (GGCN)<n>Experiments show that IBN achieves state-of-the-art forecasting performance under various missing-rate scenarios.
  arXiv  Detail & Related papers  (2025-09-09T13:27:21Z)
• Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization [52.16435732772263]
  Second-order optimization has been shown to accelerate the training of deep neural networks in many applications. However, generalization properties of second-order methods are still being debated. We show for the first time that exact Gauss-Newton (GN) updates take on a tractable form in a class of deep architectures.
  arXiv  Detail & Related papers  (2024-11-12T17:58:40Z)
• The Role of Temporal Hierarchy in Spiking Neural Networks [2.0881857682885836]
  Spiking Neural Networks (SNNs) have the potential for rich-temporal signal processing thanks to exploiting both spatial and temporal parameters. Time constants have been recently shown to have computational benefits that help reduce the overall number of parameters required in the network. To reduce the cost of optimization, architectural biases can be applied, in this case in the temporal domain. We propose to impose a hierarchy of temporal representation in the hidden layers of SNNs, highlighting that such an inductive bias improves their performance.
  arXiv  Detail & Related papers  (2024-07-26T16:00:20Z)
• The Sample Complexity of One-Hidden-Layer Neural Networks [57.6421258363243]
  We study a class of scalar-valued one-hidden-layer networks, and inputs bounded in Euclidean norm. We prove that controlling the spectral norm of the hidden layer weight matrix is insufficient to get uniform convergence guarantees. We analyze two important settings where a mere spectral norm control turns out to be sufficient.
  arXiv  Detail & Related papers  (2022-02-13T07:12:02Z)
• Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization [79.42778415729475]
  We explore an alternative solution based on explicit memorization using linear autoencoders for sequences. We show how such pretraining can better support solving hard classification tasks with long sequences. We show that the proposed approach achieves a much lower reconstruction error for long sequences and a better gradient propagation during the finetuning phase.
  arXiv  Detail & Related papers  (2020-11-05T14:57:16Z)
• Supporting Optimal Phase Space Reconstructions Using Neural Network Architecture for Time Series Modeling [68.8204255655161]
  We propose an artificial neural network with a mechanism to implicitly learn the phase spaces properties. Our approach is either as competitive as or better than most state-of-the-art strategies.
  arXiv  Detail & Related papers  (2020-06-19T21:04:47Z)
• Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
  We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
  arXiv  Detail & Related papers  (2020-02-20T15:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.