Anytime Training with Schedule-Free Spectral Optimization

• URL: http://arxiv.org/abs/2605.23061v1
• Date: Thu, 21 May 2026 21:50:22 GMT
• Title: Anytime Training with Schedule-Free Spectral Optimization
• Abstract summary: SF-NorMuon is a schedule-free spectral that matches or exceeds tuned AdamW on 125M and 772M parameter language models across $1$--$8times$ Chinchilla horizons.<n>By closing the performance gap with tuned baselines, SF-Muon makes horizon-free optimization more practical, taking a step towards truly open-ended, continual learning.
• Score: 3.843049855044539
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard neural network training relies on learning-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re-tuning as data availability changes. Schedule-Free (SF) methods address this by removing explicit schedules, yet SF-AdamW, the current state-of-the-art anytime optimizer, consistently underperforms well-tuned AdamW baselines. We propose SF-NorMuon, a schedule-free spectral optimizer that closes this gap: with a single hyperparameter configuration, SF-NorMuon matches or exceeds tuned AdamW on 125M and 772M parameter language models across $1$--$8\times$ Chinchilla horizons. On the theoretical side, we prove a stationarity guarantee for schedule-free spectral dynamics and identify weight decay at the fast iterate as essential for long-horizon stability. SF-NorMuon enables practitioners to obtain high-quality checkpoints at any point during training without committing to a horizon in advance. By closing the performance gap with tuned baselines, SF-NorMuon makes horizon-free optimization more practical, taking a step towards truly open-ended, continual learning.

Related papers

• WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training [18.733838876040537]
  We propose WSqD (Warmup with convex Square-root base and linear Decay), a learning rate schedule that replaces WSD's constant phase with a shifted inverse-square-root base while retaining the final linear.<n> Empirically, WSqD matches or outperforms carefully tuned WSD and other baselines across multiple training horizons while reusing a single peak learning rate.
  arXiv  Detail & Related papers  (2026-07-12T23:24:42Z)
• From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime [52.459513910242485]
  We develop a general statistical framework for studying optimal noise-level allocation in diffusion training.<n>We show that the optimized training schedule admits an atomic minimizer, concentrated on finitely many noise levels.<n>We then evaluate the entropic schedule in larger-scale experiments, where full schedule optimization is currently intractable.
  arXiv  Detail & Related papers  (2026-07-10T20:59:06Z)
• PrismFlow: Residual Dynamics for Flow Matching in Time-Series Generation [24.703364668945543]
  Flow Matching offers an efficient alternative to diffusion models, but practical implementations typically rely on a single finite-capacity global vector-field estimator.<n>A monolithic estimator trained with the standard $ell$ velocity-matching objective may therefore learn an overly smoothed approximation of the local transport field.<n>This estimator-level smoothing can branch-specific dynamics, leading to spectral distortion and poor mode coverage.<n>We propose PrismFlow, a new FM method with Koopman-inspired dynamical experts.
  arXiv  Detail & Related papers  (2026-05-22T07:10:20Z)
• Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization [69.03519018775779]
  Group Relative Policy Optimization is essential for aligning video diffusion models with human preferences.<n>Existing efficiency methods compromise optimization, exhibiting severe instability and failing to reach full trajectory performance.<n>We present Flash-GRPO, a single-step training framework that outperforms full trajectory training in alignment quality under low computational budgets.
  arXiv  Detail & Related papers  (2026-05-15T14:13:39Z)
• Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT [12.18095306054314]
  Conservative Supervised Fine-Tuning (ConSFT) is an optimization objective that adapts to target distributions while mitigating catastrophic forgetting.<n>We evaluate ConSFT on the LIBERO and RoboTwin benchmarks across state-of-the-art flow-matching VLAs.
  arXiv  Detail & Related papers  (2026-05-09T10:59:03Z)
• Spectral Gating Networks [65.9496901693099]
  We introduce Spectral Gating Networks (SGN) to introduce frequency-rich expressivity in feed-forward networks.<n>SGN augments a standard activation pathway with a compact spectral pathway and learnable gates that allow the model to start from a stable base behavior.<n>It consistently improves accuracy-efficiency trade-offs under comparable computational budgets.
  arXiv  Detail & Related papers  (2026-02-07T20:00:49Z)
• Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging [70.05077723711618]
  We show that language models trained at 1-32x Chinchilla scale decay with time, with the decay rate determined by the source and capacity conditions of the problem.<n>Our results suggest that weight averaging combined with simple, horizon-free step sizes offers a practical and effective anytime alternative to cosine learning rate schedules for large language model pretraining.
  arXiv  Detail & Related papers  (2026-02-03T16:24:05Z)
• Elastic Spectral State Space Models for Budgeted Inference [6.579320299248572]
  Foundation models are typically trained at a fixed computational capacity, while real-world applications require deployment across platforms with different resource constraints.<n>We propose Elastic Spectral State Space Models (ES-SSM), which require only one-time training at full capacity, but can be directly truncated into arbitrary scales for budgeted, runtime inference without retraining.<n>We demonstrate that a single ES-SSM model trained once can be truncated to provide competitive performance compared with modern Transformer and SSM baselines at similar parameter scales.
  arXiv  Detail & Related papers  (2026-01-30T02:58:19Z)
• Controlled LLM Training on Spectral Sphere [76.60985966206746]
  We introduce the textbfSpectral Sphere algorithm (SSO), which enforces strict module-wise spectral constraints on both weights and their updates.<n>We observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations.
  arXiv  Detail & Related papers  (2026-01-13T09:59:47Z)
• Unifying Quadrotor Motion Planning and Control by Chaining Different Fidelity Models [23.059638804041672]
  Unique is a unified MPC that cascades models of different fidelity within a single optimization.<n>We propose a 3D progressive smoothing schedule that morphs norm-based obstacles along the horizon.<n>Unique improves closed-loop position or velocity tracking by up to 75% compared with standard MPC and hierarchical planner-tracker baselines.
  arXiv  Detail & Related papers  (2025-12-13T18:53:34Z)
• Training-free Context-adaptive Attention for Efficient Long Context Modeling [57.703159205740185]
  Training-free Context-adaptive Attention (TCA-Attention) is a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference.<n>TCA-Attention achieves a 2.8$times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention.
  arXiv  Detail & Related papers  (2025-12-10T01:54:57Z)
• Adaptive Deadline and Batch Layered Synchronized Federated Learning [66.93447103966439]
  Federated learning (FL) enables collaborative model training across distributed edge devices while preserving data privacy, and typically operates in a round-based synchronous manner.<n>We propose ADEL-FL, a novel framework that jointly optimize per-round deadlines and user-specific batch sizes for layer-wise aggregation.
  arXiv  Detail & Related papers  (2025-05-29T19:59:18Z)
• Learning to Initialize Trajectory Optimization for Vision-Based Autonomous Flight in Unknown Environments [4.0543433786183485]
  We present a Neural-Enhanced Tray Planner (NEO-Planner) for autonomous flight in unknown environments.<n>NEO-Planner learns to predict spatial and temporal parameters for trajectories directly from raw sensor observations.<n>It reduces optimization by 20%, leading to a 26% decrease in trajectory time compared with pure optimization-based methods.
  arXiv  Detail & Related papers  (2023-09-19T15:07:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.