A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalization

• URL: http://arxiv.org/abs/2304.08914v2
• Date: Fri, 12 May 2023 10:22:25 GMT
• Title: A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalization
• Authors: Peifeng Gao, Qianqian Xu, Peisong Wen, Huiyang Shao, Zhiyong Yang, Qingming Huang
• Abstract summary: We extend original Neural Collapse Phenomenon by proving Generalized Neural Collapse hypothesis. We obtain Grassmannian Frame structure from the optimization and generalization of classification. We provide a theorem to explain Symmetric Generalization of permutation.
• Score: 91.95109845914267
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we extend original Neural Collapse Phenomenon by proving Generalized Neural Collapse hypothesis. We obtain Grassmannian Frame structure from the optimization and generalization of classification. This structure maximally separates features of every two classes on a sphere and does not require a larger feature dimension than the number of classes. Out of curiosity about the symmetry of Grassmannian Frame, we conduct experiments to explore if models with different Grassmannian Frames have different performance. As a result, we discover the Symmetric Generalization phenomenon. We provide a theorem to explain Symmetric Generalization of permutation. However, the question of why different directions of features can lead to such different generalization is still open for future investigation.

Related papers

• Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry [63.694184882697435]
  Global Covariance Pooling (GCP) has been demonstrated to improve the performance of Deep Neural Networks (DNNs) by exploiting second-order statistics of high-level representations.
  arXiv  Detail & Related papers  (2024-07-15T07:11:44Z)
• Holomorphic Floer theory I: exponential integrals in finite and infinite dimensions [0.0]
  We discuss exponential integrals and related wall-crossing structures. We develop the corresponding theories in particular generalizing Morse-Novikov theory to the holomorphic case. As a corollary, perturbative expansions of exponential integrals are resurgent.
  arXiv  Detail & Related papers  (2024-02-12T00:21:31Z)
• Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges [132.62934175555145]
  Neural Collapse (NC) is a well-known phenomenon of deep neural networks in the terminal phase of training (TPT) We propose a theoretical explanation for why continuing training can still lead to accuracy improvement on test set, even after the train accuracy has reached 100%. We refer to this newly discovered property as "non-conservative generalization"
  arXiv  Detail & Related papers  (2023-10-12T14:29:02Z)
• Ito Diffusion Approximation of Universal Ito Chains for Sampling, Optimization and Boosting [64.0722630873758]
  We consider rather general and broad class of Markov chains, Ito chains, that look like Euler-Maryama discretization of some Differential Equation. We prove the bound in $W_2$-distance between the laws of our Ito chain and differential equation.
  arXiv  Detail & Related papers  (2023-10-09T18:38:56Z)
• Entanglement asymmetry in the ordered phase of many-body systems: the Ising Field Theory [0.0]
  Global symmetries of quantum many-body systems can be spontaneously broken. In this study, we examine the entanglement asymmetry of a specific region. We also establish a field theoretic framework in the replica theory using twist operators.
  arXiv  Detail & Related papers  (2023-07-22T17:07:56Z)
• Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance [72.50214227616728]
  Interpretability methods are valuable only if their explanations faithfully describe the explained model. We consider neural networks whose predictions are invariant under a specific symmetry group.
  arXiv  Detail & Related papers  (2023-04-13T17:59:03Z)
• Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold [30.3185037354742]
  When training over normalized deep networks for classification tasks, the learned features exhibit a so-called "neural collapse" phenomenon. We show that better representations can be learned faster via feature normalization.
  arXiv  Detail & Related papers  (2022-09-19T17:26:32Z)
• Imbalance Trouble: Revisiting Neural-Collapse Geometry [27.21274327569783]
  We introduce Simplex-Encoded-Labels Interpolation (SELI) as an invariant characterization of the neural collapse phenomenon. We prove for the UFM with cross-entropy loss and vanishing regularization. We present experiments on synthetic and real datasets that confirm convergence to the SELI geometry.
  arXiv  Detail & Related papers  (2022-08-10T18:10:59Z)
• Scalars are universal: Gauge-equivariant machine learning, structured like classical physics [0.0]
  neural networks that respect the gauge symmetries -- or coordinate freedom -- of physical law. We show that it is simple to parameterize universally approximating functions that are equivariant under these symmetries. These results demonstrate theoretically that gauge-invariant deep learning models for classical physics with good scaling for large problems are feasible right now.
  arXiv  Detail & Related papers  (2021-06-11T20:51:38Z)
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators [72.62940905965267]
  Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. Are CF-INNs universal approximators for invertible functions? We prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes.
  arXiv  Detail & Related papers  (2020-06-20T02:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.