Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification

• URL: http://arxiv.org/abs/2507.21749v1
• Date: Tue, 29 Jul 2025 12:31:21 GMT
• Title: Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification
• Authors: D. Veerababu, Ashwin A. Raikar, Prasanta K. Ghosh,
• Abstract summary: This paper presents a dynamic learning rate scheduler (DLRS) algorithm that adapts the learning rate based on the loss values calculated during the training process.<n> Experiments are conducted on problems related to physics-informed neural networks (PINNs) and image classification using multilayer perceptrons and convolutional neural networks, respectively.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training neural networks can be challenging, especially as the complexity of the problem increases. Despite using wider or deeper networks, training them can be a tedious process, especially if a wrong choice of the hyperparameter is made. The learning rate is one of such crucial hyperparameters, which is usually kept static during the training process. Learning dynamics in complex systems often requires a more adaptive approach to the learning rate. This adaptability becomes crucial to effectively navigate varying gradients and optimize the learning process during the training process. In this paper, a dynamic learning rate scheduler (DLRS) algorithm is presented that adapts the learning rate based on the loss values calculated during the training process. Experiments are conducted on problems related to physics-informed neural networks (PINNs) and image classification using multilayer perceptrons and convolutional neural networks, respectively. The results demonstrate that the proposed DLRS accelerates training and improves stability.

Related papers

• HiPreNets: High-Precision Neural Networks through Progressive Training [1.5429976366871665]
  We present a framework for tuning and high-precision neural networks (HiPreNets)<n>Our approach refines a previously explored staged training technique for neural networks.<n>We discuss how to take advantage of the structure of the residuals to guide the choice loss function number parameters to use.
  arXiv  Detail & Related papers  (2025-06-18T02:12:24Z)
• Leveraging chaos in the training of artificial neural networks [3.379574469735166]
  We explore the dynamics of the neural network trajectory along training for unconventionally large learning rates.<n>We show that for a region of values of the learning rate, the GD optimization shifts away from purely exploitation-like algorithm into a regime of exploration-exploitation balance.
  arXiv  Detail & Related papers  (2025-06-10T07:41:58Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Edge of chaos as a guiding principle for modern neural network training [19.419382003562976]
  We study the role of various hyperparameters in modern neural network training algorithms in terms of the order-chaos phase diagram. In particular, we study a fully analytical feedforward neural network trained on the widely adopted Fashion-MNIST dataset.
  arXiv  Detail & Related papers  (2021-07-20T12:17:55Z)
• Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling [7.859988850911321]
  We introduce Sparse Meta Networks -- a meta-learning approach to learn online sequential adaptation algorithms for deep neural networks. We augment a deep neural network with a layer-specific fast-weight memory. We demonstrate strong performance on a variety of sequential adaptation scenarios.
  arXiv  Detail & Related papers  (2020-09-03T17:06:52Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms [0.0]
  We study the application of the Gradient-Only Line Search that is Inexact (GOLS-I) to determine the learning rate schedule for a selection of popular neural network training algorithms. GOLS-I's learning rate schedules are competitive with manually tuned learning rates, over seven optimization algorithms, three types of neural network architecture, 23 datasets and two loss functions.
  arXiv  Detail & Related papers  (2020-06-29T08:59:31Z)
• The large learning rate phase of deep learning: the catapult mechanism [50.23041928811575]
  We present a class of neural networks with solvable training dynamics. We find good agreement between our model's predictions and training dynamics in realistic deep learning settings. We believe our results shed light on characteristics of models trained at different learning rates.
  arXiv  Detail & Related papers  (2020-03-04T17:52:48Z)
• Curriculum By Smoothing [52.08553521577014]
  Convolutional Neural Networks (CNNs) have shown impressive performance in computer vision tasks such as image classification, detection, and segmentation. We propose an elegant curriculum based scheme that smoothes the feature embedding of a CNN using anti-aliasing or low-pass filters. As the amount of information in the feature maps increases during training, the network is able to progressively learn better representations of the data.
  arXiv  Detail & Related papers  (2020-03-03T07:27:44Z)
• Subset Sampling For Progressive Neural Network Learning [106.12874293597754]
  Progressive Neural Network Learning is a class of algorithms that incrementally construct the network's topology and optimize its parameters based on the training data. We propose to speed up this process by exploiting subsets of training data at each incremental training step. Experimental results in object, scene and face recognition problems demonstrate that the proposed approach speeds up the optimization procedure considerably.
  arXiv  Detail & Related papers  (2020-02-17T18:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.