OASIS: Online Activation Subspace Learning for Memory-Efficient Training

• URL: http://arxiv.org/abs/2604.09406v1
• Date: Fri, 10 Apr 2026 15:19:54 GMT
• Title: OASIS: Online Activation Subspace Learning for Memory-Efficient Training
• Abstract summary: We propose an online activation subspace learning algorithm for memory-efficient training.<n>OASIS tracks and continuously updates a low-dimensional activation subspace during training.<n>It achieves up to $2times$ lower peak memory than full fine-tuning.
• Score: 8.921166277011345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training large language models (LLMs) is constrained by memory requirements, with activations accounting for a substantial fraction of the total footprint. Existing approaches reduce memory using low-rank weight parameterizations or low-rank gradient subspaces for optimizer states, while activation memory is addressed through architectural modifications or compression schemes based on periodically updated projections. We propose OASIS, an online activation subspace learning algorithm for memory-efficient training that tracks and continuously updates a low-dimensional activation subspace during training. Intermediate activations are projected onto this evolving subspace, reducing memory without modifying forward-pass computations. The evolving activation subspace induces low-rank gradient representations, enabling both gradients and optimizer states to be maintained directly in this subspace, while a projection-aware optimizer consistently transports optimizer states across subspace updates for stable training. Across various finetuning and pretraining tasks, OASIS achieves up to $2\times$ lower peak memory than full fine-tuning while matching its performance and outperforming prior low-rank methods.

Related papers

• SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models [75.75091459417304]
  Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal understanding.<n>LVLMs incur substantial memory and latency overhead, severely limiting real-world deployment on resource-constrained devices.<n>We propose a binarization method to drastically reduce storage and computational costs.
  arXiv  Detail & Related papers  (2026-07-02T08:30:00Z)
• MemDLM: Memory-Enhanced DLM Training [56.40248490616793]
  Diffusion Language Models (DLMs) offer attractive advantages over Auto-Regressive (AR) models.<n>They suffer from a notable train-inference mismatch.<n>We propose MemDLM (Memory-Enhanced DLM), which embeds a simulated denoising process into training.
  arXiv  Detail & Related papers  (2026-03-23T17:39:56Z)
• Randomized Gradient Subspaces for Efficient Large Language Model Training [4.398170461093705]
  Training large language models (LLMs) is often bottlenecked by extreme memory demands, with states dominating the footprint.<n>Recent works mitigate this cost by projecting gradients into low-dimensional subspaces using sophisticated update strategies.<n>We find that while a small subspace captures most gradient energy, a significant portion still resides in the residual bulk.<n>We introduce a suite of randomized algorithms, GrassWalk and GrassJump, which exploit subspace and achieve state-of-the-art memory savings while improving performance on LLaMA-1B and LLaMA-7B pretraining.
  arXiv  Detail & Related papers  (2025-10-02T10:35:38Z)
• LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning [9.009523608709117]
  On-device learning is essential for personalization, privacy, and long-term adaptation in resource-constrained environments.<n>Existing activation compression methods reduce this cost but rely on repeated low-rank decompositions, introducing computational overhead.<n>We propose LANCE, a framework that performs one-shot higher-order Singular Value Decompsoition (SVD) to obtain a reusable low-rank subspace for activation projection.
  arXiv  Detail & Related papers  (2025-09-25T21:33:40Z)
• SURGEON: Memory-Adaptive Fully Test-Time Adaptation via Dynamic Activation Sparsity [30.260783715373382]
  Test-time adaptation (TTA) has emerged to improve the performance of deep models by adapting them to unlabeled target data online.<n>Yet, the significant memory cost, particularly in resource-constrained terminals, impedes the effective deployment of most backward-propagation-based TTA methods.<n>To tackle memory constraints, we introduce SURGEON, a method that substantially reduces memory cost while preserving comparable accuracy improvements.
  arXiv  Detail & Related papers  (2025-03-26T09:27:09Z)
• Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining [43.39070237124361]
  Low-rank optimization is a promising approach to enabling memory-efficient training of large language models.<n>A key challenge in these methods is selecting suitable subspaces to ensure an effective trajectory.<n>We propose importance sampling for low-rank optimization in LLM pretraining with a provable convergence guarantee.
  arXiv  Detail & Related papers  (2025-02-09T06:30:19Z)
• SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training [4.804685266333252]
  Training large language models (LLMs) is highly resource-intensive due to their massive number of parameters and the overhead of states.<n>We propose SubTrack++ that leverages Grassmannian subspace tracking combined with projection-aware gradients.<n>Our method demonstrates SOTA convergence by exploiting Grassmannian geometry, reducing pre-training wall-time by up to 65% and fine-tuning time by 36% compared to existing SOTA methods.
  arXiv  Detail & Related papers  (2025-02-03T18:13:52Z)
• Sparse Gradient Compression for Fine-Tuning Large Language Models [58.44973963468691]
  Fine-tuning large language models (LLMs) for downstream tasks has become increasingly crucial due to their widespread use and the growing availability of open-source models.<n>High memory costs associated with fine-tuning remain a significant challenge, especially as models increase in size.<n>We propose sparse compression gradient (SGC) to address these limitations.
  arXiv  Detail & Related papers  (2025-02-01T04:18:28Z)
• Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Training [78.93900796545523]
  Mini-Sequence Transformer (MsT) is a methodology for highly efficient and accurate LLM training with extremely long sequences. MsT partitions input sequences and iteratively processes mini-sequences to reduce intermediate memory usage. integrated with the huggingface library, MsT successfully extends the maximum context length of Qwen, Mistral, and Gemma-2 by 12-24x.
  arXiv  Detail & Related papers  (2024-07-22T01:52:30Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks [19.007090250576585]
  BlockLLM is an approach inspired by block coordinate descent.<n>It achieves state-of-the-art performance in both finetuning and pretraining tasks.
  arXiv  Detail & Related papers  (2024-06-25T05:45:12Z)
• Mesa: A Memory-saving Training Framework for Transformers [58.78933015299703]
  We present Mesa, a memory-saving training framework for Transformers. Mesa uses exact activations during forward pass while storing a low-precision version of activations to reduce memory consumption during training. Experiments on ImageNet, CIFAR-100 and ADE20K demonstrate that Mesa can reduce half of the memory footprints during training.
  arXiv  Detail & Related papers  (2021-11-22T11:23:01Z)
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings [89.63764845984076]
  We present Stored Embeddings for Efficient Reinforcement Learning (SEER) SEER is a simple modification of existing off-policy deep reinforcement learning methods. We show that SEER does not degrade the performance of RLizable agents while significantly saving computation and memory.
  arXiv  Detail & Related papers  (2021-03-04T08:14:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.