GradPower: Powering Gradients for Faster Language Model Pre-Training

• URL: http://arxiv.org/abs/2505.24275v1
• Date: Fri, 30 May 2025 06:49:57 GMT
• Title: GradPower: Powering Gradients for Faster Language Model Pre-Training
• Abstract summary: GradPower is a lightweight-transformation technique for accelerating language model pre-training.<n>It requires only a single-line code change and no modifications to the base's internal logic.<n>It consistently achieves lower terminal loss across diverse architectures.
• Score: 15.650619834236789
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose GradPower, a lightweight gradient-transformation technique for accelerating language model pre-training. Given a gradient vector $g=(g_i)_i$, GradPower first applies the elementwise sign-power transformation: $\varphi_p(g)=({\rm sign}(g_i)|g_i|^p)_{i}$ for a fixed $p>0$, and then feeds the transformed gradient into a base optimizer. Notably, GradPower requires only a single-line code change and no modifications to the base optimizer's internal logic, including the hyperparameters. When applied to Adam (termed AdamPower), GradPower consistently achieves lower terminal loss across diverse architectures (LLaMA, Qwen2MoE), parameter scales (66M to 2B), datasets (C4, OpenWebText), and learning-rate schedules (cosine, warmup-stable-decay). The most pronounced gains are observed when training modern mixture-of-experts models with warmup-stable-decay schedules. GradPower also integrates seamlessly with other state-of-the-art optimizers, such as Muon, yielding further improvements. Finally, we provide theoretical analyses that reveal the underlying mechanism of GradPower and highlights the influence of gradient noise.

Related papers

• Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks [29.66774607766935]
  Gradient surgery methods mitigate this issue by constructing directions from loss-specific gradients to reduce conflict before transformation.<n>Let $a_t$ denote the direction constructed by gradient surgery, $u_t$ the proposal, and $mathcalC_t$ the conflict-free gradient induced by the loss-specific gradients.<n>We propose Gradient-Update Alignment (GUA), which projects $u_t$ onto $mathcalC_t$ to obtain the aligned update $p_t$ and applies $p_t$ to the parameters.
  arXiv  Detail & Related papers  (2026-09-01T17:20:13Z)
• Grad-ELLM: Gradient-based Explanations for Decoder-only LLMs [52.15785423211181]
  Grad-ELLM is a gradient-based attribution method for decoder-only transformer-based Large Language Models.<n>We introduce two faithfulneses metrics $$-Soft-NC and $$-Soft-NS, which provide fairer comparisons.<n>Experiment results show that Grad-ELLM consistently achieves superior faithfulness than other attribution methods.
  arXiv  Detail & Related papers  (2026-01-06T15:22:39Z)
• Backward-Friendly Optimization: Training Large Language Models with Approximate Gradients under Memory Constraints [14.20716202034732]
  Full fine-tuning of Large Language Models (LLMs) is notoriously memory-intensive.<n>We introduce GradLite, a backward-friendly solution that relaxes the requirement of exact gradients.<n>We show that GradLite maintains unbiased estimates with bounded variance, ensuring convergence rates comparable to Adam.
  arXiv  Detail & Related papers  (2025-10-26T00:50:12Z)
• GradMetaNet: An Equivariant Architecture for Learning on Gradients [18.350495600116712]
  We introduce GradMetaNet, a novel architecture for learning on gradients.<n>We prove results for GradMetaNet, and show that previous approaches cannot approximate natural gradient-based functions.<n>We then demonstrate GradMetaNet's effectiveness on a diverse set of gradient-based tasks.
  arXiv  Detail & Related papers  (2025-07-02T12:22:39Z)
• Can Gradient Descent Simulate Prompting? [56.60154660021178]
  gradient updates the effects of conditioning on new information.<n> gradient descent training recovers some (and occasionally all) of prompted model performance.<n>Results suggest new avenues for long-context modeling.
  arXiv  Detail & Related papers  (2025-06-26T04:06:20Z)
• ELRA: Exponential learning rate adaption gradient descent optimization method [83.88591755871734]
  We present a novel, fast (exponential rate), ab initio (hyper-free) gradient based adaption. The main idea of the method is to adapt the $alpha by situational awareness. It can be applied to problems of any dimensions n and scales only linearly.
  arXiv  Detail & Related papers  (2023-09-12T14:36:13Z)
• Gradient Sparsification For Masked Fine-Tuning of Transformers [6.936564049727831]
  Fine-tuning pretrained self-supervised language models is widely adopted for transfer learning to downstream tasks. Gradual unfreezing makes a trade-off between the two by gradually unfreezing gradients of whole layers during training. It is not clear whether gradually unfreezing layers throughout training is optimal, compared to sparse variants of gradual unfreezing.
  arXiv  Detail & Related papers  (2023-07-19T16:13:13Z)
• Gradient Correction beyond Gradient Descent [63.33439072360198]
  gradient correction is apparently the most crucial aspect for the training of a neural network. We introduce a framework (textbfGCGD) to perform gradient correction. Experiment results show that our gradient correction framework can effectively improve the gradient quality to reduce training epochs by $sim$ 20% and also improve the network performance.
  arXiv  Detail & Related papers  (2022-03-16T01:42:25Z)
• Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant [0.0]
  We introduce an efficient gradient, called $quadratic$ $gradient$, for privacy-preserving logistic regression training. Experimental results demonstrate that the enhanced algorithms achieve significantly improved convergence speed. There is a good chance that the quadratic gradient approach could integrate first-order gradient descent/ascent algorithms with the second-order Newton-Raphson methods.
  arXiv  Detail & Related papers  (2022-01-26T09:44:13Z)
• Primer: Searching for Efficient Transformers for Language Modeling [79.2677566332444]
  Training and inference costs of large Transformer models have grown rapidly and become expensive. Here we aim to reduce the costs of Transformers by searching for a more efficient variant. We identify an architecture, named Primer, that has a smaller training cost than the original Transformer.
  arXiv  Detail & Related papers  (2021-09-17T17:50:39Z)
• Tom: Leveraging trend of the observed gradients for faster convergence [0.0]
  Tom is a novel variant of Adam that takes into account the trend observed for the gradients in the landscape in the loss traversed by the neural network. Tom outperforms Adagrad, Adadelta, RMSProp and Adam in terms of both accuracy and has a faster convergence.
  arXiv  Detail & Related papers  (2021-09-07T20:19:40Z)
• Adapting Stepsizes by Momentumized Gradients Improves Optimization and Generalization [89.66571637204012]
  textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing.
  arXiv  Detail & Related papers  (2021-06-22T03:13:23Z)
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training [59.160154997555956]
  We present GradInit, an automated and architecture method for initializing neural networks. It is based on a simple agnostic; the variance of each network layer is adjusted so that a single step of SGD or Adam results in the smallest possible loss value. It also enables training the original Post-LN Transformer for machine translation without learning rate warmup.
  arXiv  Detail & Related papers  (2021-02-16T11:45:35Z)
• Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets [71.05306664267832]
  Adaptive algorithms perform gradient updates using the history of gradients and are ubiquitous in training deep neural networks. In this paper we analyze a variant of OptimisticOA algorithm for nonconcave minmax problems. Our experiments show that adaptive GAN non-adaptive gradient algorithms can be observed empirically.
  arXiv  Detail & Related papers  (2019-12-26T22:10:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.