WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

• URL: http://arxiv.org/abs/2607.10959v1
• Date: Sun, 12 Jul 2026 23:24:42 GMT
• Title: WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training
• Abstract summary: We propose WSqD (Warmup with convex Square-root base and linear Decay), a learning rate schedule that replaces WSD's constant phase with a shifted inverse-square-root base while retaining the final linear.<n> Empirically, WSqD matches or outperforms carefully tuned WSD and other baselines across multiple training horizons while reusing a single peak learning rate.
• Score: 18.733838876040537
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard learning rate schedules such as cosine annealing are tied to a fixed training horizon, limiting their ability to accommodate post hoc horizon extension. Warmup-stable-decay (WSD) partially addresses this issue by maintaining a long constant-rate phase before a short linear cooldown, allowing training to resume from a pre-decay checkpoint. However, its peak learning rate is still tuned based on the original training horizon and can become suboptimal when training is extended. Motivated by stochastic convex optimization, we propose WSqD (Warmup with Square-root base and linear Decay), a learning rate schedule that replaces WSD's constant stable phase with a shifted inverse-square-root base while retaining the final linear cooldown. In the stochastic convex setting, WSqD provably attains the minimax-optimal $O(1/\sqrt{T})$ last-iterate convergence rate. Importantly, its base learning rate schedule is horizon-independent, and the training horizon is needed only to determine when to begin the final cooldown. Empirically, on language-model pretraining using the SlimPajama corpus, WSqD matches or outperforms carefully tuned WSD and other baselines across multiple training horizons while reusing a single peak learning rate.

Related papers

• Train Smarter, Not Harder: Switching Signal-Guided Training in Active Learning [1.215621649989295]
  We show that retraining is most useful in early rounds, while fine-tuning becomes safer once the model trajectory stabilizes.<n>We propose HybridAL, an adaptive training schedule that monitors an online stabilization signal and switches from retraining to fine-tuning after sustained stabilization.
  arXiv  Detail & Related papers  (2026-09-06T19:45:17Z)
• Learning to Solve Generative ODEs Beyond the Linear Span [50.13853710831612]
  We propose SpanLift, a lightweight neural solver that augments scalar-coefficient updates with a spatial residual operator.<n>Across pixel-space diffusion, latent flow matching, and precipitation nowcasting, SpanLift achieves state-of-the-art few-step sampling.
  arXiv  Detail & Related papers  (2026-06-07T15:22:13Z)
• Anytime Training with Schedule-Free Spectral Optimization [3.843049855044539]
  SF-NorMuon is a schedule-free spectral that matches or exceeds tuned AdamW on 125M and 772M parameter language models across $1$--$8times$ Chinchilla horizons.<n>By closing the performance gap with tuned baselines, SF-Muon makes horizon-free optimization more practical, taking a step towards truly open-ended, continual learning.
  arXiv  Detail & Related papers  (2026-05-21T21:50:22Z)
• TempoFit: Plug-and-Play Layer-Wise Temporal KV Memory for Long-Horizon Vision-Language-Action Manipulation [16.28432866472846]
  We present TempoFit, a training-free temporal retrofit that upgrades frozen Vision-Language-Action policies through state-level memory.<n>Our key insight is that prefix attention K/V already form a model-native, content-addressable runtime state.<n>On LIBERO-LONG, TempoFit improves strong pretrained backbones by up to +4.0% average success rate while maintaining near-real-time latency.
  arXiv  Detail & Related papers  (2026-03-08T14:17:25Z)
• Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging [70.05077723711618]
  We show that language models trained at 1-32x Chinchilla scale decay with time, with the decay rate determined by the source and capacity conditions of the problem.<n>Our results suggest that weight averaging combined with simple, horizon-free step sizes offers a practical and effective anytime alternative to cosine learning rate schedules for large language model pretraining.
  arXiv  Detail & Related papers  (2026-02-03T16:24:05Z)
• Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling [75.36692892951018]
  Increasing the batch size during training is a promising strategy to accelerate large language model pretraining.<n>This work develops a principled framework for batch-size scheduling.<n>It introduces Seesaw: whenever a standard scheduler would halve the learning rate, Seesaw instead multiplies it by $1/sqrt2$ and doubles the batch size.
  arXiv  Detail & Related papers  (2025-10-16T14:17:38Z)
• Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective [66.80315289020487]
  Warmup-Stable-Decay (WSD) schedule uses a constant learning rate to produce a main branch of iterates that can continue indefinitely without a pre-specified compute budget.<n>We show that pretraining loss exhibits a river valley landscape, which resembles a deep valley with a river at its bottom.<n>Inspired by the theory, we introduce WSD-S, a variant of WSD that reuses previous checkpoints' decay phases and keeps only one main branch.
  arXiv  Detail & Related papers  (2024-10-07T16:49:39Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Do We Need Zero Training Loss After Achieving Zero Training Error? [76.44358201918156]
  We propose a direct solution called emphflooding that intentionally prevents further reduction of the training loss when it reaches a reasonably small value. We experimentally show that flooding improves performance and, as a byproduct, induces a double descent curve of the test loss.
  arXiv  Detail & Related papers  (2020-02-20T12:50:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.