An Unconstrained Layer-Peeled Perspective on Neural Collapse

• URL: http://arxiv.org/abs/2110.02796v1
• Date: Wed, 6 Oct 2021 14:18:47 GMT
• Title: An Unconstrained Layer-Peeled Perspective on Neural Collapse
• Authors: Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, Weijie J. Su
• Abstract summary: We introduce a surrogate model called the unconstrained layer-peeled model (ULPM) We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer. We show that our results also hold during the training of neural networks in real-world tasks when explicit regularization or weight decay is not used.
• Score: 20.75423143311858
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural collapse is a highly symmetric geometric pattern of neural networks that emerges during the terminal phase of training, with profound implications on the generalization performance and robustness of the trained networks. To understand how the last-layer features and classifiers exhibit this recently discovered implicit bias, in this paper, we introduce a surrogate model called the unconstrained layer-peeled model (ULPM). We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer. Moreover, we show that the ULPM with the cross-entropy loss has a benign global landscape for its loss function, which allows us to prove that all the critical points are strict saddle points except the global minimizers that exhibit the neural collapse phenomenon. Empirically, we show that our results also hold during the training of neural networks in real-world tasks when explicit regularization or weight decay is not used.

Related papers

• The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features [0.0]
  Deep neural networks exhibit a simple structure in their final layer features and weights, commonly referred to as neural collapse. Recent findings indicate that such a structure is generally not optimal in the deep unconstrained feature model. This is attributed to a low-rank bias induced by regularization, which favors solutions with lower-rank than those typically associated with deep neural collapse.
  arXiv  Detail & Related papers  (2024-10-30T16:20:39Z)
• Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features [33.703796571991745]
  Recent findings reveal that overparameterized deep neural networks, trained beyond zero training, exhibit a distinctive structural pattern at the final layer. These results indicate that the final-layer outputs in such networks display minimal within-class variations.
  arXiv  Detail & Related papers  (2024-02-29T06:02:45Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data [12.225207401994737]
  We show that complex systems with massive amounts of parameters exhibit the same structural properties when training until convergence. In particular, it has been observed that the last-layer features collapse to their class-means. Our results demonstrate the convergence of the last-layer features and classifiers to a geometry consisting of vectors.
  arXiv  Detail & Related papers  (2023-01-01T16:29:56Z)
• Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold [30.3185037354742]
  When training over normalized deep networks for classification tasks, the learned features exhibit a so-called "neural collapse" phenomenon. We show that better representations can be learned faster via feature normalization.
  arXiv  Detail & Related papers  (2022-09-19T17:26:32Z)
• On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features [38.05002597295796]
  Collapselayers collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) An intriguing empirical phenomenon has been widely observed in the last-layers and features of deep neural networks for tasks.
  arXiv  Detail & Related papers  (2022-03-02T17:00:18Z)
• Extended Unconstrained Features Model for Exploring Deep Neural Collapse [59.59039125375527]
  Recently, a phenomenon termed "neural collapse" (NC) has been empirically observed in deep neural networks. Recent papers have shown that minimizers with this structure emerge when optimizing a simplified "unconstrained features model" In this paper, we study the UFM for the regularized MSE loss, and show that the minimizers' features can be more structured than in the cross-entropy case.
  arXiv  Detail & Related papers  (2022-02-16T14:17:37Z)
• The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks [51.1848572349154]
  neural network models that perfectly fit noisy data can generalize well to unseen test data. We consider interpolating two-layer linear neural networks trained with gradient flow on the squared loss and derive bounds on the excess risk.
  arXiv  Detail & Related papers  (2021-08-25T22:01:01Z)
• Topological obstructions in neural networks learning [67.8848058842671]
  We study global properties of the loss gradient function flow. We use topological data analysis of the loss function and its Morse complex to relate local behavior along gradient trajectories with global properties of the loss surface.
  arXiv  Detail & Related papers  (2020-12-31T18:53:25Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.