Leveraging chaos in the training of artificial neural networks

• URL: http://arxiv.org/abs/2506.08523v1
• Date: Tue, 10 Jun 2025 07:41:58 GMT
• Title: Leveraging chaos in the training of artificial neural networks
• Authors: Pedro Jiménez-González, Miguel C. Soriano, Lucas Lacasa,
• Abstract summary: We explore the dynamics of the neural network trajectory along training for unconventionally large learning rates.<n>We show that for a region of values of the learning rate, the GD optimization shifts away from purely exploitation-like algorithm into a regime of exploration-exploitation balance.
• Score: 3.379574469735166
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional algorithms to optimize artificial neural networks when confronted with a supervised learning task are usually exploitation-type relaxational dynamics such as gradient descent (GD). Here, we explore the dynamics of the neural network trajectory along training for unconventionally large learning rates. We show that for a region of values of the learning rate, the GD optimization shifts away from purely exploitation-like algorithm into a regime of exploration-exploitation balance, as the neural network is still capable of learning but the trajectory shows sensitive dependence on initial conditions -- as characterized by positive network maximum Lyapunov exponent --. Interestingly, the characteristic training time required to reach an acceptable accuracy in the test set reaches a minimum precisely in such learning rate region, further suggesting that one can accelerate the training of artificial neural networks by locating at the onset of chaos. Our results -- initially illustrated for the MNIST classification task -- qualitatively hold for a range of supervised learning tasks, learning architectures and other hyperparameters, and showcase the emergent, constructive role of transient chaotic dynamics in the training of artificial neural networks.

Related papers

• Peer-to-Peer Learning Dynamics of Wide Neural Networks [10.179711440042123]
  We provide an explicit characterization of the learning dynamics of wide neural networks trained using distributed gradient descent (DGD) algorithms.<n>Our results leverage both recent in neural tangent kernel (NTK) theory and extensive previous work on distributed learning and consensus tasks.
  arXiv  Detail & Related papers  (2024-09-23T17:57:58Z)
• Dynamical stability and chaos in artificial neural network trajectories along training [3.379574469735166]
  We study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network. We find hints of regular and chaotic behavior depending on the learning rate regime. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning.
  arXiv  Detail & Related papers  (2024-04-08T17:33:11Z)
• NeuralFastLAS: Fast Logic-Based Learning from Raw Data [54.938128496934695]
  Symbolic rule learners generate interpretable solutions, however they require the input to be encoded symbolically. Neuro-symbolic approaches overcome this issue by mapping raw data to latent symbolic concepts using a neural network. We introduce NeuralFastLAS, a scalable and fast end-to-end approach that trains a neural network jointly with a symbolic learner.
  arXiv  Detail & Related papers  (2023-10-08T12:33:42Z)
• Enhanced quantum state preparation via stochastic prediction of neural network [0.8287206589886881]
  In this paper, we explore an intriguing avenue for enhancing algorithm effectiveness through exploiting the knowledge blindness of neural network. Our approach centers around a machine learning algorithm utilized for preparing arbitrary quantum states in a semiconductor double quantum dot system. By leveraging prediction generated by the neural network, we are able to guide the optimization process to escape local optima.
  arXiv  Detail & Related papers  (2023-07-27T09:11:53Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Dynamic Analysis of Nonlinear Civil Engineering Structures using Artificial Neural Network with Adaptive Training [2.1202971527014287]
  In this study, artificial neural networks are developed with adaptive training algorithms. The networks can successfully predict the time-history response of the shear frame and the rock structure to real ground motion records.
  arXiv  Detail & Related papers  (2021-11-21T21:14:48Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Edge of chaos as a guiding principle for modern neural network training [19.419382003562976]
  We study the role of various hyperparameters in modern neural network training algorithms in terms of the order-chaos phase diagram. In particular, we study a fully analytical feedforward neural network trained on the widely adopted Fashion-MNIST dataset.
  arXiv  Detail & Related papers  (2021-07-20T12:17:55Z)
• Stochastic Markov Gradient Descent and Training Low-Bit Neural Networks [77.34726150561087]
  We introduce Gradient Markov Descent (SMGD), a discrete optimization method applicable to training quantized neural networks. We provide theoretical guarantees of algorithm performance as well as encouraging numerical results.
  arXiv  Detail & Related papers  (2020-08-25T15:48:15Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.