On the Cone Effect in the Learning Dynamics

• URL: http://arxiv.org/abs/2503.16316v2
• Date: Sun, 13 Apr 2025 10:24:49 GMT
• Title: On the Cone Effect in the Learning Dynamics
• Authors: Zhanpeng Zhou, Yongyi Yang, Jie Ren, Mahito Sugiyama, Junchi Yan,
• Abstract summary: We take an empirical perspective to study the learning dynamics of neural networks in real-world settings.<n>Our key findings reveal a two-phase learning process: i) in Phase I, the eNTK evolves significantly, signaling the rich regime, and ii) in Phase II, the eNTK keeps evolving but is constrained in a narrow space.
• Score: 57.02319387815831
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding the learning dynamics of neural networks is a central topic in the deep learning community. In this paper, we take an empirical perspective to study the learning dynamics of neural networks in real-world settings. Specifically, we investigate the evolution process of the empirical Neural Tangent Kernel (eNTK) during training. Our key findings reveal a two-phase learning process: i) in Phase I, the eNTK evolves significantly, signaling the rich regime, and ii) in Phase II, the eNTK keeps evolving but is constrained in a narrow space, a phenomenon we term the cone effect. This two-phase framework builds on the hypothesis proposed by Fort et al. (2020), but we uniquely identify the cone effect in Phase II, demonstrating its significant performance advantages over fully linearized training.

Related papers

• Contrastive Learning in Memristor-based Neuromorphic Systems [55.11642177631929]
  Spiking neural networks have become an important family of neuron-based models that sidestep many of the key limitations facing modern-day backpropagation-trained deep networks. In this work, we design and investigate a proof-of-concept instantiation of contrastive-signal-dependent plasticity (CSDP), a neuromorphic form of forward-forward-based, backpropagation-free learning.
  arXiv  Detail & Related papers  (2024-09-17T04:48:45Z)
• Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks [59.38765771221084]
  We present a physiologically inspired speech recognition architecture compatible and scalable with deep learning frameworks. We show end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronising neural activity to improve recognition performance.
  arXiv  Detail & Related papers  (2024-04-22T09:40:07Z)
• The dynamic interplay between in-context and in-weight learning in humans and neural networks [15.744573869783972]
  We show that "in-context learning" (ICL) can equip neural networks with fundamentally different learning properties that can coexist with their native IWL. Our work shows how emergent ICL can equip neural networks with fundamentally different learning properties that can coexist with their native IWL.
  arXiv  Detail & Related papers  (2024-02-13T18:55:27Z)
• Three Mechanisms of Feature Learning in a Linear Network [0.34530027457862006]
  We present an exact solution for the learning dynamics of a one-hidden-layer linear network, with one-dimensional data, across any finite width.<n>We identify three novel prototype mechanisms specific to the feature learning regime.<n>Our findings are substantiated with empirical evidence showing that these mechanisms also manifest in deep nonlinear networks handling real-world tasks.
  arXiv  Detail & Related papers  (2024-01-13T14:21:46Z)
• Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics [6.349503549199403]
  We provide a comprehensive framework for the learning process of deep wide neural networks.<n>By characterizing the diffusive phase, our work sheds light on representational drift in the brain.
  arXiv  Detail & Related papers  (2023-09-08T18:00:01Z)
• Phase Diagram of Initial Condensation for Two-layer Neural Networks [4.404198015660192]
  We present a phase diagram of initial condensation for two-layer neural networks. Our phase diagram serves to provide a comprehensive understanding of the dynamical regimes of neural networks.
  arXiv  Detail & Related papers  (2023-03-12T03:55:38Z)
• Critical Learning Periods for Multisensory Integration in Deep Networks [112.40005682521638]
  We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early phases of training. We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations.
  arXiv  Detail & Related papers  (2022-10-06T23:50:38Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• The Three Stages of Learning Dynamics in High-Dimensional Kernel Methods [14.294357100033954]
  We consider the training dynamics of flow on least-squares objectives, which is a limiting dynamics of SGD trained neural networks. We show that SGD progressively learns more complex functions and that there is a "deep gradient" phenomenon.
  arXiv  Detail & Related papers  (2021-11-13T18:29:36Z)
• The large learning rate phase of deep learning: the catapult mechanism [50.23041928811575]
  We present a class of neural networks with solvable training dynamics. We find good agreement between our model's predictions and training dynamics in realistic deep learning settings. We believe our results shed light on characteristics of models trained at different learning rates.
  arXiv  Detail & Related papers  (2020-03-04T17:52:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.