ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models

• URL: http://arxiv.org/abs/2605.19095v1
• Date: Mon, 18 May 2026 20:31:49 GMT
• Title: ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
• Abstract summary: We present a learning-rate-free and schedule-free method (ScheduleFree+) for training large language models which greatly outperforms Warmup-Stable-Decay (WSD) schedules.<n>We also demonstrate that Schedule-Free Learning is most effective for long duration training, and at 1000 tokens per parameter, it outperforms SOTA schedules by 31%.
• Score: 10.720913594892117
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Schedule-Free Learning has shown promise as a practical anytime training method for machine learning, showing success across dozens of standard benchmark problems. However, strong performance for LLM training has only been demonstrated at small scales. We identify a number of fixes necessary to scale up Schedule-Free Learning to larger batch sizes and model sizes, and present a learning-rate-free and schedule-free method (ScheduleFree+) for training large language models which greatly outperforms Warmup-Stable-Decay (WSD) schedules. We also demonstrate that Schedule-Free Learning is most effective for long duration training, and at 1000 tokens per parameter, it outperforms SOTA schedules by 31%. Schedule-Free Learning provides a theoretical foundation for the use of model averaging and checkpoint merging during pretraining.

Related papers

• NanoNet: Parameter-Efficient Learning with Label-Scarce Supervision for Lightweight Text Mining Model [51.055122269052696]
  NanoNet is a novel framework for lightweight text mining that implements parameter-efficient learning with limited supervision.<n>The entire process leverages parameter-efficient learning, reducing training costs and minimizing supervision requirements, ultimately yielding a lightweight model for downstream inference.
  arXiv  Detail & Related papers  (2026-02-05T08:31:57Z)
• Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging [70.05077723711618]
  We show that language models trained at 1-32x Chinchilla scale decay with time, with the decay rate determined by the source and capacity conditions of the problem.<n>Our results suggest that weight averaging combined with simple, horizon-free step sizes offers a practical and effective anytime alternative to cosine learning rate schedules for large language model pretraining.
  arXiv  Detail & Related papers  (2026-02-03T16:24:05Z)
• Dynamics of Learning: Generative Schedules from Latent ODEs [0.14323566945483496]
  We present a new learning rate scheduler that models the training performance of neural networks as a dynamical system.<n>Our method is computationally efficient, generalization-agnostic, and can easily be layered on top of ML experiment-tracking platforms.
  arXiv  Detail & Related papers  (2025-09-27T02:20:18Z)
• A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules [67.87680482844884]
  We present an empirical law that describes how the pretraining loss of large language models evolves under different learning rate schedules.<n>Our proposed law takes a multi-power form, combining a power law based on the sum of learning rates and additional power laws to account for a loss reduction effect induced by learning rate decay.
  arXiv  Detail & Related papers  (2025-03-17T04:36:45Z)
• Anytime Continual Learning for Open Vocabulary Classification [15.228942895385432]
  AnytimeCL problem aims to break away from batch training and rigid models. We propose a dynamic weighting between predictions of a partially fine-tuned model and a fixed open vocabulary model. Our methods are validated with experiments that test flexibility of learning and inference.
  arXiv  Detail & Related papers  (2024-09-13T03:34:37Z)
• Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review [50.78587571704713]
  Learn-Focus-Review (LFR) is a dynamic training approach that adapts to the model's learning progress.<n>LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset.<n>Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy.
  arXiv  Detail & Related papers  (2024-09-10T00:59:18Z)
• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations [62.132347451049455]
  Scale has become a main ingredient in obtaining strong machine learning models. In this work, we argue that scale and training research has been needlessly complex due to reliance on the cosine schedule. We show that weight averaging yields improved performance along the training trajectory, without additional training costs, across different scales.
  arXiv  Detail & Related papers  (2024-05-28T17:33:54Z)
• Fast Exact Unlearning for In-Context Learning Data for LLMs [30.06631665962119]
  We show that for large language models we can efficiently unlearn "fine-tuning data"<n>We show that accurate in-context learning can be done with quantized k-means, which allows for effectively constant time unlearning operations.
  arXiv  Detail & Related papers  (2024-02-01T16:43:04Z)
• How to Scale Your EMA [20.94711634514331]
  We provide a scaling rule for optimization in the presence of a model EMA. We show the rule's validity where the model EMA contributes to the optimization of the target model. For Self-Supervised Learning, we enable training of BYOL up to batch size 24,576 without sacrificing performance.
  arXiv  Detail & Related papers  (2023-07-25T20:33:48Z)
• Learning Rate Perturbation: A Generic Plugin of Learning Rate Schedule towards Flatter Local Minima [40.70374106466073]
  We propose a generic learning rate schedule plugin called LEArning Rate Perturbation (LEAP) LEAP can be applied to various learning rate schedules to improve the model training by introducing a certain perturbation to the learning rate. We conduct extensive experiments which show that training with LEAP can improve the performance of various deep learning models on diverse datasets.
  arXiv  Detail & Related papers  (2022-08-25T05:05:18Z)
• Automated Learning Rate Scheduler for Large-batch Training [24.20872850681828]
  Large-batch training has been essential in leveraging large-scale datasets and models in deep learning. It often requires a specially designed learning rate (LR) schedule to achieve a comparable level of performance as in smaller batch training. We propose an automated LR scheduling algorithm which is effective for neural network training with a large batch size under the given epoch budget.
  arXiv  Detail & Related papers  (2021-07-13T05:23:13Z)
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets [106.79387235014379]
  EarlyBERT is a general computationally-efficient training algorithm applicable to both pre-training and fine-tuning of large-scale language models. We are the first to identify structured winning tickets in the early stage of BERT training, and use them for efficient training. EarlyBERT easily achieves comparable performance to standard BERT with 3545% less training time.
  arXiv  Detail & Related papers  (2020-12-31T20:38:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.