Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysis

• URL: http://arxiv.org/abs/2001.03371v1
• Date: Fri, 10 Jan 2020 10:17:08 GMT
• Title: Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysis
• Abstract summary: plateau phenomenon, wherein the loss value stops decreasing during the process of learning, has been reported by various researchers. In this paper, using statistical mechanical formulation, we clarified the relationship between the plateau phenomenon and the statistical property of the data learned.
• Score: 5.299781502918034
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The plateau phenomenon, wherein the loss value stops decreasing during the process of learning, has been reported by various researchers. The phenomenon is actively inspected in the 1990s and found to be due to the fundamental hierarchical structure of neural network models. Then the phenomenon has been thought as inevitable. However, the phenomenon seldom occurs in the context of recent deep learning. There is a gap between theory and reality. In this paper, using statistical mechanical formulation, we clarified the relationship between the plateau phenomenon and the statistical property of the data learned. It is shown that the data whose covariance has small and dispersed eigenvalues tend to make the plateau phenomenon inconspicuous.

Related papers

• Explaining Grokking and Information Bottleneck through Neural Collapse Emergence [33.22494588674352]
  We present a unified explanation of grokking and the information bottleneck principle through the lens of neural collapse.<n>We show that the contraction of population within-class variance is a key factor underlying both grokking and information bottleneck.<n>By analyzing the dynamics of neural collapse, we show that distinct time scales between fitting the training set and the progression of neural collapse account for the behavior of the late-phase phenomena.
  arXiv  Detail & Related papers  (2025-09-25T07:17:41Z)
• A dynamic view of some anomalous phenomena in SGD [0.0]
  It has been observed that over-parametrized neural networks exhibit a double descent' phenomenon.<n>Another anomalous phenomenon is that of itgrokking wherein two regimes of descent are interrupted by a third regime wherein the mean loss remains almost constant.<n>This note presents a plausible explanation for these and related phenomena by using the theory of two time scale approximation, applied to the continuous time limit dynamics.
  arXiv  Detail & Related papers  (2025-05-03T09:01:10Z)
• Rethinking Benign Overfitting in Two-Layer Neural Networks [2.486161976966064]
  We refine the feature-noise data model by incorporating class-dependent heterogeneous noise and re-examine the overfitting phenomenon in neural networks. Our findings reveal that neural networks can leverage "data noise", previously deemed harmful, to learn implicit features that improve the classification accuracy for long-tailed data.
  arXiv  Detail & Related papers  (2025-02-17T15:20:04Z)
• Losing dimensions: Geometric memorization in generative diffusion [10.573546162574235]
  We extend the theory of memorization in generative diffusion to manifold-supported data. Our theoretical and experimental findings indicate that different subspaces are lost due to memorization effects at different critical times and dataset sizes. Perhaps counterintuitively, we find that, under some conditions, subspaces of higher variance are lost first due to memorization effects.
  arXiv  Detail & Related papers  (2024-10-11T11:31:20Z)
• Multiple Descents in Unsupervised Learning: The Role of Noise, Domain Shift and Anomalies [14.399035468023161]
  We study the presence of double descent in unsupervised learning, an area that has received little attention and is not yet fully understood. We use synthetic and real data and identify model-wise, epoch-wise, and sample-wise double descent for various applications.
  arXiv  Detail & Related papers  (2024-06-17T16:24:23Z)
• Disentangling the Causes of Plasticity Loss in Neural Networks [55.23250269007988]
  We show that loss of plasticity can be decomposed into multiple independent mechanisms. We show that a combination of layer normalization and weight decay is highly effective at maintaining plasticity in a variety of synthetic nonstationary learning tasks.
  arXiv  Detail & Related papers  (2024-02-29T00:02:33Z)
• Deep Machine Learning Reconstructing Lattice Topology with Strong Thermal Fluctuations [10.55785051508181]
  Deep convolutional neural network (CNN) reconstructed lattice topology in presence of strong thermal fluctuations and unbalanced data. Our scheme distinguishes from the previous ones that might require the knowledge on the node dynamics. We unveil the generalization of CNN on dealing with the instances evolved from the unlearnt initial spin configurations.
  arXiv  Detail & Related papers  (2022-08-08T13:28:21Z)
• Memorization-Dilation: Modeling Neural Collapse Under Label Noise [10.134749691813344]
  During the terminal phase of training a deep neural network, the feature embedding of all examples of the same class tend to collapse to a single representation. Empirical evidence suggests that the memorization of noisy data points leads to a degradation (dilation) of the neural collapse. Our proofs reveal why label smoothing, a modification of cross-entropy empirically observed to produce a regularization effect, leads to improved generalization in classification tasks.
  arXiv  Detail & Related papers  (2022-06-11T13:40:37Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks [51.1848572349154]
  neural network models that perfectly fit noisy data can generalize well to unseen test data. We consider interpolating two-layer linear neural networks trained with gradient flow on the squared loss and derive bounds on the excess risk.
  arXiv  Detail & Related papers  (2021-08-25T22:01:01Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)
• Vulnerability Under Adversarial Machine Learning: Bias or Variance? [77.30759061082085]
  We investigate the effect of adversarial machine learning on the bias and variance of a trained deep neural network. Our analysis sheds light on why the deep neural networks have poor performance under adversarial perturbation. We introduce a new adversarial machine learning algorithm with lower computational complexity than well-known adversarial machine learning strategies.
  arXiv  Detail & Related papers  (2020-08-01T00:58:54Z)
• On Disentangled Representations Learned From Correlated Data [59.41587388303554]
  We bridge the gap to real-world scenarios by analyzing the behavior of the most prominent disentanglement approaches on correlated data. We show that systematically induced correlations in the dataset are being learned and reflected in the latent representations. We also demonstrate how to resolve these latent correlations, either using weak supervision during training or by post-hoc correcting a pre-trained model with a small number of labels.
  arXiv  Detail & Related papers  (2020-06-14T12:47:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.