FOCUS: First Order Concentrated Updating Scheme

• URL: http://arxiv.org/abs/2501.12243v1
• Date: Tue, 21 Jan 2025 16:03:42 GMT
• Title: FOCUS: First Order Concentrated Updating Scheme
• Abstract summary: Large language models (LLMs) demonstrate remarkable performance, and improving their pre-training process appears to be key to enhancing their capabilities further.<n>We show that when gradient query noise is high relative to the valley's sharpness, Adam's performance falls behind that of Signum because Adam reduces the effective step size too drastically.<n>We develop FOCUS, which enhances Signum by incorporating attraction toward moving averaged parameters, allowing it to handle noise better while maintaining larger step sizes.
• Score: 9.278468089636547
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) demonstrate remarkable performance, and improving their pre-training process appears to be key to enhancing their capabilities further. Based on the documented success of Adam, learning rate decay, and weight decay, we hypothesize that the pre-training loss landscape features a narrowing valley structure. Through experiments with synthetic loss functions, we discover that when gradient query noise is high relative to the valley's sharpness, Adam's performance falls behind that of Signum because Adam reduces the effective step size too drastically. This observation led us to develop FOCUS, an optimizer that enhances Signum by incorporating attraction toward moving averaged parameters, allowing it to handle noise better while maintaining larger step sizes. In training GPT-2, FOCUS proves to be more stable than Signum and faster than Adam. These results suggest that gradient noise may be an underappreciated limiting factor in LLM training, and FOCUS offers promising solutions.

Related papers

• Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates [14.645982276552592]
  gradient descent (SGD) performs significantly worse than adaptives such as Adam in pre-training Large Language Models.<n>In this work, we attribute a large part of the discrepancy to SGD's inability to sustain learning rates comparable to Adam's much larger effective learning rates.<n>We show that simple clipping mechanisms that stabilize SGD at large learning rates enable it to recover most of Adam's performance.
  arXiv  Detail & Related papers  (2026-05-18T03:09:50Z)
• NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation [66.36912000442608]
  NoisyRollout is a simple yet effective data augmentation method.<n>It mixes training trajectories from both clean and moderately distorted images.<n>It achieves state-of-the-art performance among open-source RL-tuned models.
  arXiv  Detail & Related papers  (2025-04-17T16:10:13Z)
• SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training [60.9776082805359]
  Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks, yet their training remains highly resource-intensive and susceptible to training instability.<n>This paper presents a comprehensive investigation into gradient spikes observed during LLM training, revealing their prevalence across multiple architectures and datasets.<n>We propose Spike-Aware Adam with Momentum Reset, a novel designed to counteract gradient spikes through momentum reset and spike-aware clipping.
  arXiv  Detail & Related papers  (2025-01-12T15:21:22Z)
• Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? [40.94505326255136]
  Low-rank training has emerged as a promising approach for reducing memory usage in training Large Language Models. We propose a new plug-and-play training framework for LLMs called Fira, as the first attempt to achieve this goal. We show that Fira outperforms both LoRA and GaLore, achieving performance that is comparable to or even better than full-rank training.
  arXiv  Detail & Related papers  (2024-10-02T14:58:27Z)
• Efficient Continual Pre-training by Mitigating the Stability Gap [68.49269649759005]
  We study the behavior of Large Language Models (LLMs) during continual pre-training. We propose three effective strategies to enhance LLM performance within a fixed compute budget. Our strategies improve the average medical task performance of the OpenLlama-3B model from 36.2% to 40.7% with only 40% of the original training budget.
  arXiv  Detail & Related papers  (2024-06-21T02:28:37Z)
• Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed [83.8485684139678]
  Methods with adaptive stepsizes, such as AdaGrad and Adam, are essential for training modern Deep Learning models.<n>We show that AdaGrad/Adam methods can have bad high-probability convergence if the noise is heavy-tailed.
  arXiv  Detail & Related papers  (2024-06-06T18:49:10Z)
• Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE [62.13435256279566]
  Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks. However, their large size makes their inference slow and computationally expensive. We show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer.
  arXiv  Detail & Related papers  (2023-10-28T04:07:58Z)
• StochGradAdam: Accelerating Neural Networks Training with Stochastic Gradient Sampling [0.0]
  We introduce StochGradAdam, a novel extension of the Adam algorithm, incorporating gradient sampling techniques. StochGradAdam achieves comparable or superior performance to Adam, even when using fewer gradient updates per iteration. The results suggest that this approach is particularly effective for large-scale models and datasets.
  arXiv  Detail & Related papers  (2023-10-25T22:45:31Z)
• Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be [16.170888329408353]
  We show that the behavior of Adam with large batches is similar to sign descent with momentum. We present evidence thatity and heavy-tailed noise are not major factors in the performance gap between SGD and Adam.
  arXiv  Detail & Related papers  (2023-04-27T05:41:13Z)
• Noise Augmentation Is All You Need For FGSM Fast Adversarial Training: Catastrophic Overfitting And Robust Overfitting Require Different Augmentation [72.87038663276554]
  Adversarial training (AT) and its variants are the most effective approaches for obtaining adversarially robust models. In this work, we show that single-step F GSMAT is not as hard as suggested in the past line of work: noise augmentation is all you need for (F GSM) fast AT.
  arXiv  Detail & Related papers  (2022-02-11T07:37:55Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks [64.7563588124004]
  We argue for the existence of the "break-even" point on this trajectory. We show that using a large learning rate in the initial phase of training reduces the variance of the gradient. We also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers.
  arXiv  Detail & Related papers  (2020-02-21T22:55:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.