Randomized Gradient Subspaces for Efficient Large Language Model Training

• URL: http://arxiv.org/abs/2510.01878v1
• Date: Thu, 02 Oct 2025 10:35:38 GMT
• Title: Randomized Gradient Subspaces for Efficient Large Language Model Training
• Abstract summary: Training large language models (LLMs) is often bottlenecked by extreme memory demands, with states dominating the footprint.<n>Recent works mitigate this cost by projecting gradients into low-dimensional subspaces using sophisticated update strategies.<n>We find that while a small subspace captures most gradient energy, a significant portion still resides in the residual bulk.<n>We introduce a suite of randomized algorithms, GrassWalk and GrassJump, which exploit subspace and achieve state-of-the-art memory savings while improving performance on LLaMA-1B and LLaMA-7B pretraining.
• Score: 4.398170461093705
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Training large language models (LLMs) is often bottlenecked by extreme memory demands, with optimizer states dominating the footprint. Recent works mitigates this cost by projecting gradients into low-dimensional subspaces using sophisticated update strategies. In this paper, we analyze the dynamics of gradient space and its underlying subspaces. We find that while a small subspace captures most gradient energy, a significant portion still resides in the residual bulk; moreover, the influence of the core subspace diminishes over time and in deeper layers. We also observe that the gradient space exhibits near-flat curvature, calling for algorithms that explicitly account for this geometry. Motivated by these insights, we introduce a suite of randomized algorithms, GrassWalk and GrassJump, which exploit subspace and achieve state-of-the-art memory savings while improving performance on LLaMA-1B and LLaMA-7B pretraining.

Related papers

• OASIS: Online Activation Subspace Learning for Memory-Efficient Training [8.921166277011345]
  We propose an online activation subspace learning algorithm for memory-efficient training.<n>OASIS tracks and continuously updates a low-dimensional activation subspace during training.<n>It achieves up to $2times$ lower peak memory than full fine-tuning.
  arXiv  Detail & Related papers  (2026-04-10T15:19:54Z)
• AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning [8.698253005940503]
  We propose Activation-Guided Zeroth-Order optimization (AGZO)<n>Unlike prior methods, AGZO extracts a compact, activation-informed subspace on the fly during the forward pass and restricts perturbations to this low-rank subspace.<n>AGZO consistently outperforms state-of-the-art ZO baselines and significantly narrows the performance gap with first-order fine-tuning.
  arXiv  Detail & Related papers  (2026-01-24T02:28:15Z)
• Dimer-Enhanced Optimization: A First-Order Approach to Escaping Saddle Points in Neural Network Training [5.9408311406202285]
  Dimer method is a first-order technique that constructs two closely spaced points to probe the local geometry of a potential energy surface.<n>Inspired by its use in molecular dynamics simulations for locating saddle points, we propose Dimer-Enhanced Optimization.<n>DEO guides the away from saddle points and flat regions, enhancing training efficiency with non-stepwise updates.
  arXiv  Detail & Related papers  (2025-07-26T14:57:32Z)
• Large Language Models Encode Semantics in Low-Dimensional Linear Subspaces [31.401762286885656]
  Understanding the latent space geometry of large language models (LLMs) is key to their behavior and alignment.<n>We conduct a large-scale study in 11 empirical models across 6 scientific topics.
  arXiv  Detail & Related papers  (2025-07-13T17:03:25Z)
• SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting [68.00007494819798]
  Continual learning requires a model to learn multiple tasks in sequence while maintaining both stability:preserving knowledge from previously learned tasks, and plasticity:effectively learning new tasks.<n> Gradient projection has emerged as an effective and popular paradigm in CL, where it partitions the gradient space of previously learned tasks into two subspaces.<n>New tasks are learned effectively within the minor subspace, thereby reducing interference with previously acquired knowledge.<n>Existing Gradient Projection methods struggle to achieve an optimal balance between plasticity and stability, as it is hard to appropriately partition the gradient space.
  arXiv  Detail & Related papers  (2025-05-28T13:57:56Z)
• I3S: Importance Sampling Subspace Selection for Low-Rank Optimization in LLM Pretraining [50.89661053183944]
  Low-rank optimization has emerged as a promising approach to enabling memory-efficient training of large language models (LLMs)<n>Existing low-rank optimization methods typically project gradients onto a low-rank subspace, reducing the memory cost of storing states.<n>We propose importance sampling subspace selection (I3S) for low-rank optimization, which theoretically offers a comparable convergence rate to the dominant subspace approach.
  arXiv  Detail & Related papers  (2025-02-09T06:30:19Z)
• TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs [58.19080159470868]
  We propose a novel low-rank ZO estimator, TeZO, which captures the low-rankness across both the model and temporal dimension.<n>Specifically, we represent ZO perturbations along the temporal dimension as a 3D tensor and employ Canonical Polyadic Decomposition (CPD) to extract each low-rank 2D matrix.
  arXiv  Detail & Related papers  (2025-01-31T11:34:03Z)
• Zeroth-Order Fine-Tuning of LLMs in Random Subspaces [63.10833446782114]
  As language models grow in size, memory demands for backpropagation increase.<n>Zeroth-order (ZO) optimization methods offer a memory-efficient alternative.<n>In this paper, we propose Subspace Zero-order optimization to address the challenges posed by posed by high dimensionality perturbations.
  arXiv  Detail & Related papers  (2024-10-11T17:01:43Z)
• Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients [86.40635601953446]
  We introduce Q-Galore, a novel approach that substantially reduces memory usage by combining quantization and low-rank projection. We demonstrate that Q-Galore achieves highly competitive performance with exceptional memory efficiency.
  arXiv  Detail & Related papers  (2024-07-11T08:42:58Z)
• Class Gradient Projection For Continual Learning [99.105266615448]
  Catastrophic forgetting is one of the most critical challenges in Continual Learning (CL) We propose Class Gradient Projection (CGP), which calculates the gradient subspace from individual classes rather than tasks.
  arXiv  Detail & Related papers  (2023-11-25T02:45:56Z)
• Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank? [26.055358499719027]
  We propose the "Look-back Gradient Multiplier" (LBGM) algorithm, which exploits this low-rank property to enable gradient recycling. We analytically characterize the convergence behavior of LBGM, revealing the nature of the trade-off between communication savings and model performance. We show that LBGM is a general plug-and-play algorithm that can be used standalone or stacked on top of existing sparsification techniques for distributed model training.
  arXiv  Detail & Related papers  (2022-02-01T09:05:32Z)
• Intermediate Layer Optimization for Inverse Problems using Deep Generative Models [86.29330440222199]
  ILO is a novel optimization algorithm for solving inverse problems with deep generative models. We empirically show that our approach outperforms state-of-the-art methods introduced in StyleGAN-2 and PULSE for a wide range of inverse problems.
  arXiv  Detail & Related papers  (2021-02-15T06:52:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.