Enhancing LLM Training via Spectral Clipping

• URL: http://arxiv.org/abs/2603.14315v1
• Date: Sun, 15 Mar 2026 10:16:40 GMT
• Title: Enhancing LLM Training via Spectral Clipping
• Abstract summary: SPECTRA is a framework for post-spectral clipping of updates to enforce spectral-norm constraints.<n>We show that SPECTRA uniformly improves loss for various validations, including AdamW, Signum, and AdEMAMix, with the best-performing variants achieving state-of-the-art results.
• Score: 28.270053200038586
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While spectral-based optimizers like Muon operate directly on the spectrum of updates, standard adaptive methods such as AdamW do not account for the global spectral structure of weights and gradients, leaving them vulnerable to two empirical issues in large language model (LLM) training: (i) the optimizer updates can have large spectral norms, potentially destabilizing training and degrading generalization; (ii) stochastic gradient noise can exhibit sparse spectral spikes, with a few dominant singular values much larger than the rest. We propose SPECTRA, a general framework addressing these by (i) post-spectral clipping of updates to enforce spectral-norm constraints; (ii) optional pre-spectral clipping of gradients to suppress spectral noise spikes. We prove that post-clipping constitutes a Composite Frank-Wolfe method with spectral-norm constraints and weight regularization, recovering Frobenius and $\ell_{\infty}$-norm regularization with SGD-based and sign-based methods. We further analyze how pre-clipping mitigates sparse spectral spikes. We propose efficient soft spectral clipping via Newton-Schulz iterations, avoiding expensive SVD. Experiments on LLM pretraining show SPECTRA uniformly improves validation loss for various optimizers, including AdamW, Signum, and AdEMAMix, with the best-performing variants achieving state-of-the-art results. Models trained with SPECTRA exhibit smaller weight norms, confirming the link between spectral clipping and regularization.

Related papers

• Spectrally Deconfounded Gradient Boosting [10.803532361460741]
  We develop a nonlinear spectral deconfounding framework for gradient boosting.<n>We show that deconfounding is not achieved by the spectral loss alone.<n>We also extend the method to general likelihoods and nonlinear confounding.
  arXiv  Detail & Related papers  (2026-07-10T12:51:42Z)
• SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneity [57.84199526590779]
  Federated Learning (FL) is fundamentally challenged by statistical heterogeneity, where non-identically distributed (non-IID) data induces client drift that hampers global convergence.<n>In this work, we revisit client drift from a novel frequency-domain perspective and uncover a critical Spectral Bias of Drift.<n>Motivated by this, we propose SpecGradFilter, a unified Spectral Filtering Gradient Framework that tames heterogeneity by suppressing discordant low-frequency signals.
  arXiv  Detail & Related papers  (2026-07-05T09:23:35Z)
• Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters [48.3175117923623]
  Most gradient clipping rules treat all parameters as heavy vectors and ignore the matrix structure of modern vectors.<n>We show empirically that data outliers often amplify only a small number of leading singular values.<n>Motivated by this phenomenon, we propose clipping, which stabilizes by singular values.
  arXiv  Detail & Related papers  (2026-05-12T09:24:59Z)
• Generalization from Low- to Moderate-Resolution Spectra with Neural Networks for Stellar Parameter Estimation: A Case Study with DESI [33.83813864639123]
  Cross-survey generalization is a critical challenge in stellar spectral analysis, particularly in cases such as transferring from low to moderate-resolution surveys.<n>We investigate this problem using pre-trained models, focusing on simple neural networks such as multilayer perceptrons (MLPs)<n>Specifically, we pre-trains on either LRS or their embeddings and fine-tune them for application to DESI stellar spectra.<n>We find that pre-trained on LAMOST LRS achieve strong performance, even without fine-tuning, and that modest fine-tuning with DESI spectra further improves the results.
  arXiv  Detail & Related papers  (2026-02-16T18:58:47Z)
• Spectral Gating Networks [65.9496901693099]
  We introduce Spectral Gating Networks (SGN) to introduce frequency-rich expressivity in feed-forward networks.<n>SGN augments a standard activation pathway with a compact spectral pathway and learnable gates that allow the model to start from a stable base behavior.<n>It consistently improves accuracy-efficiency trade-offs under comparable computational budgets.
  arXiv  Detail & Related papers  (2026-02-07T20:00:49Z)
• On the Spectral Flattening of Quantized Embeddings [25.64641307046705]
  Training Large Language Models at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic heavy-tailed spectral nature of linguistic data.<n>This work not only quantifies the spectral sensitivity of LLMs but also establishes spectral fidelity as a necessary condition for stable low-bit optimization.
  arXiv  Detail & Related papers  (2026-02-01T02:21:53Z)
• Natural Spectral Fusion: p-Exponent Cyclic Scheduling and Early Decision-Boundary Alignment in First-Order Optimization [11.323131201168572]
  We propose Natural Spectral Fusion (NSF): reframing training as controllable spectral coverage and information fusion.<n>NSF has two core principles: treating the balances as a spectral controller that dynamically low- and high-frequency information.<n>We show that cyclic scheduling consistently reduces test error and demonstrates distinct convergence behavior.
  arXiv  Detail & Related papers  (2025-09-05T00:00:00Z)
• Towards Anomaly-Aware Pre-Training and Fine-Tuning for Graph Anomaly Detection [59.042018542376596]
  Graph anomaly detection (GAD) has garnered increasing attention in recent years, yet remains challenging due to two key factors.<n>Anomaly-Aware Pre-Training and Fine-Tuning (APF) is a framework to mitigate the challenges in GAD.<n> Comprehensive experiments on 10 benchmark datasets validate the superior performance of APF in comparison to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-19T09:57:35Z)
• Gradient Normalization Provably Benefits Nonconvex SGD under Heavy-Tailed Noise [60.92029979853314]
  We investigate the roles of gradient normalization and clipping in ensuring the convergence of Gradient Descent (SGD) under heavy-tailed noise. Our work provides the first theoretical evidence demonstrating the benefits of gradient normalization in SGD under heavy-tailed noise. We introduce an accelerated SGD variant incorporating gradient normalization and clipping, further enhancing convergence rates under heavy-tailed noise.
  arXiv  Detail & Related papers  (2024-10-21T22:40:42Z)
• Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space [54.13671100638092]
  Holistic Physics Mixer (HPM) is a framework for integrating spectral and physical information in a unified space.<n>We show that HPM consistently outperforms state-of-the-art methods in both accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2024-10-15T08:19:39Z)
• Spectral Adapter: Fine-Tuning in Spectral Space [45.72323731094864]
  We study the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure. We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion.
  arXiv  Detail & Related papers  (2024-05-22T19:36:55Z)
• MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction [148.26195175240923]
  We propose a novel Transformer-based method, Multi-stage Spectral-wise Transformer (MST++) for efficient spectral reconstruction. In the NTIRE 2022 Spectral Reconstruction Challenge, our approach won the First place.
  arXiv  Detail & Related papers  (2022-04-17T02:39:32Z)
• Hyperspectral Image Denoising Using Non-convex Local Low-rank and Sparse Separation with Spatial-Spectral Total Variation Regularization [49.55649406434796]
  We propose a novel non particular approach to robust principal component analysis for HSI denoising. We develop accurate approximations to both rank and sparse components. Experiments on both simulated and real HSIs demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-01-08T11:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.