Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold

• URL: http://arxiv.org/abs/2604.00230v1
• Date: Tue, 31 Mar 2026 20:52:42 GMT
• Title: Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold
• Abstract summary: Neural collapse -- convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but dynamics governing its onset remain poorly characterised.<n>We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*.<n>Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn*; (3) weight decay defines a three-regime phase diagram -- too little, an optimal range is fastest, and too much prevents collapse;
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised. We identify a simple and predictive regularity: NC occurs when the mean feature norm reaches a model-dataset-specific critical value, fn*, that is largely invariant to training conditions. This value concentrates tightly within each (model, dataset) pair (CV < 8%); training dynamics primarily affect the rate at which fn approaches fn*, rather than the value itself. In standard training trajectories, the crossing of fn below fn* consistently precedes NC onset, providing a practical predictor with a mean lead time of 62 epochs (MAE 24 epochs). A direct intervention experiment confirms fn* is a stable attractor of the gradient flow -- perturbations to feature scale are self-corrected during training, with convergence to the same value regardless of direction (p>0.2). Completing the (architecture)x(dataset) grid reveals the paper's strongest result: ResNet-20 on MNIST gives fn* = 5.867 -- a +458% architecture effect versus only +68% on CIFAR-10. The grid is strongly non-additive; fn* cannot be decomposed into independent architecture and dataset contributions. Four structural regularities emerge: (1) depth has a non-monotonic effect on collapse speed; (2) activation jointly determines both collapse speed and fn*; (3) weight decay defines a three-regime phase diagram -- too little slows, an optimal range is fastest, and too much prevents collapse; (4) width monotonically accelerates collapse while shifting fn* by at most 13%. These results establish feature-norm dynamics as an actionable diagnostic for predicting NC timing, suggesting that norm-threshold behaviour is a general mechanism underlying delayed representational reorganisation in deep networks.

Related papers

• When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay [6.625270757109195]
  We show that a single quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect.<n>This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes.
  arXiv  Detail & Related papers  (2026-09-08T17:46:17Z)
• NICE: Scale-Stable Perturbations for Graph Neural Network Explanations via Noise Corruption [67.57304101271096]
  Element-wise Masking (EM) suppresses edge-induced messages toward zero.<n>Noise Corruption (NC) perturbs each message through matched-norm random-direction corruption.<n>NICE, a Noise Corruption-based explanation framework, learns a Restoration Boundary (SRB) under NC-induced uncertainty.
  arXiv  Detail & Related papers  (2026-08-17T03:08:14Z)
• Robust Neural Tucker Factorization with Bias Correction and Adaptive Initialization [1.104960878651584]
  High-dimensional incomplete (HDI) tensors are widely used in traffic and climate applications, but sparse observations make accurate completion difficult.<n>This paper proposes a simple yet effective neural Tucker factorization model with Kaiming initialization and bias correction (KaBiN) for HDI tensor completion.<n> Experiments on three real-world HDI tensor datasets show that KaBiN achieves better performance than the original NeuTucF, while introducing minimal computational overhead.
  arXiv  Detail & Related papers  (2026-06-15T08:17:52Z)
• Deformable Gaussian Occupancy: Decoupling Rigid and Nonrigid Motion with Factorized Distillation [81.44955493339835]
  DeGO is a deformable Gaussian occupancy framework that unifies decoupled Gaussian deformation with factorized 4D foundation-model distillation.<n> Experiments on the Occ3D-NuScenes benchmark demonstrate that our method achieves state-of-the-art performance under weak supervision.
  arXiv  Detail & Related papers  (2026-05-27T15:09:55Z)
• Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Models [0.0]
  We propose a novel framework that elevates discrete pruning to a continuous kinetic integral over the depth manifold of a model.<n>Experiments on Qwen-2.5-7B and LLaMA-3-8B improve performance in the high-sparsity regime around 40%.<n>RKU attains 13.34% accuracy on GSM8K at 40% sparsity, outperforming the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-09T15:47:53Z)
• Recovering Physical Dynamics from Discrete Observations via Intrinsic Differential Consistency [12.266189649117003]
  We replace local supervision with a global structural constraint.<n>We train a time-conditioned secant velocity field whose deviation from this property serves two purposes.<n>As a training regularizer, it confines the hypothesis space to flows that compose consistently across temporal scales.
  arXiv  Detail & Related papers  (2026-05-08T20:24:32Z)
• Unveiling the Mechanism of Continuous Representation Full-Waveform Inversion: A Wave Based Neural Tangent Kernel Framework [52.632217600064344]
  Full-waveform inversion (FWI) estimates physical parameters in the wave equation from limited measurements.<n>FWI methods are limited by their notorious sensitivity to the accuracy of the initial models.<n>Recent progress in continuous representation FWI (CR-FWI) demonstrates that representing parameter models with a coordinate-based neural network, such as implicit neural representation (INR), can mitigate the dependence on initial models.
  arXiv  Detail & Related papers  (2026-03-23T03:22:33Z)
• K-GMRF: Kinetic Gauss-Markov Random Field for First-Principles Covariance Tracking on Lie Groups [8.489406212619164]
  We propose K-GMRF, an online, training-free framework for covariance tracking.<n>Our method interprets observations as torques driving a latent angular velocity, propagated via a structure-preserving symplectic integrator.<n>We theoretically prove that this second-order dynamics achieves zero steady-state error under constant rotation.
  arXiv  Detail & Related papers  (2026-03-20T03:16:36Z)
• Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks [52.153950303594684]
  We propose a decoupled kinetic paradigm inspired by Alternating Gradient Flow (AGF)<n>AGF accurately captures the network's structural "kinetic utility"<n>We design a hybrid routing framework that decouples AGF-guided offline structural search from online execution via zero-cost physical priors.
  arXiv  Detail & Related papers  (2026-03-12T18:19:21Z)
• DeNOTS: Stable Deep Neural ODEs for Time Series [0.99450247450967]
  CDEs provide a way to process the temporal evolution of irregular time series.<n>We propose scaling the integration time horizon to increase NFEs and "deepen" the model.<n>We also propose a way to stabilize the dynamics via Negative Feedback.
  arXiv  Detail & Related papers  (2024-08-15T09:49:37Z)
• FedNAR: Federated Optimization with Normalized Annealing Regularization [54.42032094044368]
  We explore the choices of weight decay and identify that weight decay value appreciably influences the convergence of existing FL algorithms. We develop Federated optimization with Normalized Annealing Regularization (FedNAR), a plug-in that can be seamlessly integrated into any existing FL algorithms.
  arXiv  Detail & Related papers  (2023-10-04T21:11:40Z)
• Intensity Profile Projection: A Framework for Continuous-Time Representation Learning for Dynamic Networks [50.2033914945157]
  We present a representation learning framework, Intensity Profile Projection, for continuous-time dynamic network data. The framework consists of three stages: estimating pairwise intensity functions, learning a projection which minimises a notion of intensity reconstruction error. Moreoever, we develop estimation theory providing tight control on the error of any estimated trajectory, indicating that the representations could even be used in quite noise-sensitive follow-on analyses.
  arXiv  Detail & Related papers  (2023-06-09T15:38:25Z)
• Reliable Graph Neural Networks via Robust Aggregation [1.7188280334580197]
  Perturbations targeting the graph structure have proven to be extremely effective in reducing the performance of Graph Neural Networks (GNNs) Traditional defenses such as adversarial training do not seem to be able to improve robustness. This work is motivated by the observation that adversarially injected edges effectively can be viewed as additional samples to a node's neighborhood aggregation function.
  arXiv  Detail & Related papers  (2020-10-29T14:55:20Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.