Space Alignment Matters: The Missing Piece for Inducing Neural Collapse in Long-Tailed Learning

• URL: http://arxiv.org/abs/2512.07844v1
• Date: Tue, 25 Nov 2025 07:38:40 GMT
• Title: Space Alignment Matters: The Missing Piece for Inducing Neural Collapse in Long-Tailed Learning
• Authors: Jinping Wang, Zhiqiang Gao, Zhiwu Xie,
• Abstract summary: We show that under class-balanced conditions, the class feature means and classifier weights spontaneously align into a simplexangular tight frame (ETF)<n>In long-tailed regimes, however, severe sample imbalance tends to prevent the emergence of the NC phenomenon, resulting in poor generalization performance.<n>We propose three explicit alignment strategies that plug-and-play into existing long-tail methods without architectural change.
• Score: 8.526510873614034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent studies on Neural Collapse (NC) reveal that, under class-balanced conditions, the class feature means and classifier weights spontaneously align into a simplex equiangular tight frame (ETF). In long-tailed regimes, however, severe sample imbalance tends to prevent the emergence of the NC phenomenon, resulting in poor generalization performance. Current efforts predominantly seek to recover the ETF geometry by imposing constraints on features or classifier weights, yet overlook a critical problem: There is a pronounced misalignment between the feature and the classifier weight spaces. In this paper, we theoretically quantify the harm of such misalignment through an optimal error exponent analysis. Built on this insight, we propose three explicit alignment strategies that plug-and-play into existing long-tail methods without architectural change. Extensive experiments on the CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT datasets consistently boost examined baselines and achieve the state-of-the-art performances.

Related papers

• Neural Collapse in Test-Time Adaptation [12.636904518839303]
  Test-Time Adaptation (TTA) enhances robustness to out-of-distribution data by updating the model online during inference.<n>Recently, Neural Collapse (NC) has been proposed as an emergent geometric property of deep neural networks (DNNs)<n>We identify that the performance degradation stems from sample-wise misalignment in adaptation which exacerbates under larger distribution shifts.<n>We propose NCTTA, a novel feature-classifier alignment method with hybrid targets to mitigate the impact of unreliable pseudo-labels.
  arXiv  Detail & Related papers  (2025-12-11T08:34:58Z)
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data [52.737775129027575]
  We show that gradient flow on a two-layer ReLU network for classifying orthogonally separable data provably exhibits Neural Collapse (NC)<n>We reveal the role of the implicit bias of the training dynamics in facilitating the emergence of NC.
  arXiv  Detail & Related papers  (2025-10-24T01:36:19Z)
• Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP) [0.0]
  CLOP is a novel semi-supervised loss function designed to prevent neural collapse by promoting the formation of linear subspaces among class embeddings. We show that CLOP enhances performance, providing greater stability across different learning rates and batch sizes.
  arXiv  Detail & Related papers  (2024-03-27T15:48:16Z)
• Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features [33.703796571991745]
  Recent findings reveal that overparameterized deep neural networks, trained beyond zero training, exhibit a distinctive structural pattern at the final layer. These results indicate that the final-layer outputs in such networks display minimal within-class variations.
  arXiv  Detail & Related papers  (2024-02-29T06:02:45Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges [132.62934175555145]
  Neural Collapse (NC) is a well-known phenomenon of deep neural networks in the terminal phase of training (TPT) We propose a theoretical explanation for why continuing training can still lead to accuracy improvement on test set, even after the train accuracy has reached 100%. We refer to this newly discovered property as "non-conservative generalization"
  arXiv  Detail & Related papers  (2023-10-12T14:29:02Z)
• A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks [44.31777384413466]
  Graph neural networks (GNNs) have become increasingly popular for classification tasks on graph-structured data. In this paper, we focus on node-wise classification and explore the feature evolution through the lens of the "Neural Collapse" phenomenon. We show that even an "optimistic" mathematical model requires that the graphs obey a strict structural condition in order to possess a minimizer with exact collapse.
  arXiv  Detail & Related papers  (2023-07-04T23:03:21Z)
• Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data [12.225207401994737]
  We show that complex systems with massive amounts of parameters exhibit the same structural properties when training until convergence. In particular, it has been observed that the last-layer features collapse to their class-means. Our results demonstrate the convergence of the last-layer features and classifiers to a geometry consisting of vectors.
  arXiv  Detail & Related papers  (2023-01-01T16:29:56Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning [97.81549071978789]
  We propose Attraction-Repulsion-Balanced Loss (ARB-Loss) to balance the different components of the gradients. We perform experiments on the large-scale classification and segmentation datasets and our ARB-Loss can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-04-19T08:23:23Z)
• Extended Unconstrained Features Model for Exploring Deep Neural Collapse [59.59039125375527]
  Recently, a phenomenon termed "neural collapse" (NC) has been empirically observed in deep neural networks. Recent papers have shown that minimizers with this structure emerge when optimizing a simplified "unconstrained features model" In this paper, we study the UFM for the regularized MSE loss, and show that the minimizers' features can be more structured than in the cross-entropy case.
  arXiv  Detail & Related papers  (2022-02-16T14:17:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.