Dynamics of Learning: Generative Schedules from Latent ODEs

• URL: http://arxiv.org/abs/2509.23052v1
• Date: Sat, 27 Sep 2025 02:20:18 GMT
• Title: Dynamics of Learning: Generative Schedules from Latent ODEs
• Authors: Matt L. Sampson, Peter Melchior,
• Abstract summary: We present a new learning rate scheduler that models the training performance of neural networks as a dynamical system.<n>Our method is computationally efficient, generalization-agnostic, and can easily be layered on top of ML experiment-tracking platforms.
• Score: 0.14323566945483496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The learning rate schedule is one of the most impactful aspects of neural network optimization, yet most schedules either follow simple parametric functions or react only to short-term training signals. None of them are supported by a comprehensive temporal view of how well neural networks actually train. We present a new learning rate scheduler that models the training performance of neural networks as a dynamical system. It leverages training runs from a hyperparameter search to learn a latent representation of the training process. Given current training metrics, it predicts the future learning rate schedule with the best long-term validation performance. Our scheduler generalizes beyond previously observed training dynamics and creates specialized schedules that deviate noticeably from common parametric functions. It achieves SOTA results for image classification with CNN and ResNet models as well as for next-token prediction with a transformer model. The trained models are located in flatter regions of the loss landscape and thus provide better generalization than those trained with other schedules. Our method is computationally efficient, optimizer-agnostic, and can easily be layered on top of ML experiment-tracking platforms. An implementation of our scheduler will be made available after acceptance.

Related papers

• Context-Aware Inference via Performance Forecasting in Decentralized Learning Networks [0.0]
  In decentralized learning networks, predictions from many participants are combined to generate a network inference.<n>We develop a model that uses machine learning to forecast the performance of predictions by models at each epoch in a time series.
  arXiv  Detail & Related papers  (2025-10-07T20:30:21Z)
• A Dynamical Model of Neural Scaling Laws [79.59705237659547]
  We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
  arXiv  Detail & Related papers  (2024-02-02T01:41:38Z)
• DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning [5.2319020651074215]
  We propose a Curricumum-guided Contrastive Learning framework for neural Predictor (DCLP) Our method simplifies the contrastive task by designing a novel curriculum to enhance the stability of unlabeled training data distribution. We experimentally demonstrate that DCLP has high accuracy and efficiency compared with existing predictors.
  arXiv  Detail & Related papers  (2023-02-25T08:16:21Z)
• Online Evolutionary Neural Architecture Search for Multivariate Non-Stationary Time Series Forecasting [72.89994745876086]
  This work presents the Online Neuro-Evolution-based Neural Architecture Search (ONE-NAS) algorithm. ONE-NAS is a novel neural architecture search method capable of automatically designing and dynamically training recurrent neural networks (RNNs) for online forecasting tasks. Results demonstrate that ONE-NAS outperforms traditional statistical time series forecasting methods.
  arXiv  Detail & Related papers  (2023-02-20T22:25:47Z)
• Boosted Dynamic Neural Networks [53.559833501288146]
  A typical EDNN has multiple prediction heads at different layers of the network backbone. To optimize the model, these prediction heads together with the network backbone are trained on every batch of training data. Treating training and testing inputs differently at the two phases will cause the mismatch between training and testing data distributions. We formulate an EDNN as an additive model inspired by gradient boosting, and propose multiple training techniques to optimize the model effectively.
  arXiv  Detail & Related papers  (2022-11-30T04:23:12Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones [80.662250618795]
  This paper presents a new curriculum learning approach for the efficient training of visual backbones (e.g., vision Transformers) As an off-the-shelf method, it reduces the wall-time training cost of a wide variety of popular models by >1.5x on ImageNet-1K/22K without sacrificing accuracy.
  arXiv  Detail & Related papers  (2022-11-17T17:38:55Z)
• Pretraining Graph Neural Networks for few-shot Analog Circuit Modeling and Design [68.1682448368636]
  We present a supervised pretraining approach to learn circuit representations that can be adapted to new unseen topologies or unseen prediction tasks. To cope with the variable topological structure of different circuits we describe each circuit as a graph and use graph neural networks (GNNs) to learn node embeddings. We show that pretraining GNNs on prediction of output node voltages can encourage learning representations that can be adapted to new unseen topologies or prediction of new circuit level properties.
  arXiv  Detail & Related papers  (2022-03-29T21:18:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.