Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices

• URL: http://arxiv.org/abs/2606.19528v1
• Date: Wed, 17 Jun 2026 19:20:06 GMT
• Title: Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices
• Abstract summary: Fine-tuning of Large Language Models (LLMs) on an end-user's data offers personalized experiences while keeping data private.<n>This paper introduces a suite of complementary techniques to reduce memory footprint without sacrificing model quality.<n> Experiments on Llama-3.2 3B and Qwen-2.5 3B demonstrate up to $26times$ and $28times$ reduction in peak memory.
• Score: 13.58362631516482
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Fine-tuning of Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) on an end-user's data offers personalized experiences while keeping data private, but faces severe memory constraints on consumer hardware. Peak memory during fine-tuning often exceeds device limits, especially for models with billions of parameters and long-context training data. This paper introduces a suite of complementary techniques to reduce memory footprint without sacrificing model quality: (1) base model quantization with on-the-fly dequantization, (2) memory-efficient checkpointing combining selective activation caching and disk offloading, (3) softmax approximation using semantically relevant token subsets, and (4) logits masking. Experiments on Llama-3.2 3B and Qwen-2.5 3B demonstrate up to $26\times$ and $28\times$ reduction in peak memory, enabling fine-tuning on resource-constrained devices.

Related papers

• Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation [1.2559585990041289]
  LARS (Low-memory Activation-Rank Subspace) is a novel adaptation framework that decouples memory consumption from sequence length.<n>We show that LARS reduces the memory footprint by an average of 33.54% on GPUs and 51.95% on CPUs in comparison to LoRA across reasoning, understanding and long-context datasets.
  arXiv  Detail & Related papers  (2026-04-03T17:05:33Z)
• Memory-Efficient Structured Backpropagation for On-Device LLM Fine-Tuning [10.913120072779193]
  On-device fine-tuning enables privacy-preserving personalization of large language models.<n>Mobile devices impose severe memory constraints, typically 6--12GB shared across all workloads.<n>We propose Memory-efficient Structured Backpropagation (MeSP)<n>MeSP reduces peak memory from 361MB to 136MB for Qwen2.5-0.5B, enabling fine-tuning scenarios previously infeasible.
  arXiv  Detail & Related papers  (2026-02-13T16:24:33Z)
• On-Device Fine-Tuning via Backprop-Free Zeroth-Order Optimization [27.237134457089194]
  Memory-efficient zeroth-order optimization (MeZO) alleviates this bottleneck.<n>This paper first provides a theoretical estimate of the relative model sizes that can be accommodated under BP and MeZO training.<n>We then numerically validate the analysis, demonstrating that MeZO exhibits accuracy advantages under on-device memory constraints.
  arXiv  Detail & Related papers  (2025-11-14T14:46:29Z)
• CARVQ: Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding Compression [0.4104352271917982]
  Large Language Models (LLMs) rely on a large number of parameters for token embedding, leading to substantial storage requirements and memory footprints.<n>We introduce CARVQ, a post-training novel Corrective Adaptor combined with group Residual Vector Quantization.<n>CarVQ mimics the original model embedding to compress to approximately 1.6 bits without requiring specialized hardware to support lower-bit storage.
  arXiv  Detail & Related papers  (2025-10-14T17:00:13Z)
• Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics [64.62231094774211]
  Statefuls (e.g., Adam) maintain auxiliary information even 2x the model size in order to achieve optimal convergence.<n>SOLO enables Adam-styles to maintain quantized states with precision as low as 3 bits, or even 2 bits.<n>SOLO can thus be seamlessly applied to Adam-styles, leading to substantial memory savings with minimal accuracy loss.
  arXiv  Detail & Related papers  (2025-05-01T06:47:45Z)
• A Universal Framework for Compressing Embeddings in CTR Prediction [68.27582084015044]
  We introduce a Model-agnostic Embedding Compression (MEC) framework that compresses embedding tables by quantizing pre-trained embeddings.<n>Our approach consists of two stages: first, we apply popularity-weighted regularization to balance code distribution between high- and low-frequency features.<n> Experiments on three datasets reveal that our method reduces memory usage by over 50x while maintaining or improving recommendation performance.
  arXiv  Detail & Related papers  (2025-02-21T10:12:34Z)
• Sparse Gradient Compression for Fine-Tuning Large Language Models [58.44973963468691]
  Fine-tuning large language models (LLMs) for downstream tasks has become increasingly crucial due to their widespread use and the growing availability of open-source models.<n>High memory costs associated with fine-tuning remain a significant challenge, especially as models increase in size.<n>We propose sparse compression gradient (SGC) to address these limitations.
  arXiv  Detail & Related papers  (2025-02-01T04:18:28Z)
• Memory Layers at Scale [67.00854080570979]
  This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale.<n>On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the budget, as well as mixture-of-expert models when matched for both compute and parameters.<n>We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters.
  arXiv  Detail & Related papers  (2024-12-12T23:56:57Z)
• Fast Matrix Multiplications for Lookup Table-Quantized LLMs [58.11584672945781]
  FLUTE is a flexible lookup table engine for LUT-quantized LLMs.<n>At batch sizes 32 and quantization group size of 128, the FLUTE kernel can be 2-4x faster than existing GEMM kernels.
  arXiv  Detail & Related papers  (2024-07-15T17:55:42Z)
• DPZero: Private Fine-Tuning of Language Models without Backpropagation [49.365749361283704]
  We introduce DPZero, a novel private zeroth-order algorithm with nearly dimension-independent rates. The memory efficiency of DPZero is demonstrated in privately fine-tuning RoBERTa and OPT on several downstream tasks.
  arXiv  Detail & Related papers  (2023-10-14T18:42:56Z)
• Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems [27.419109620575313]
  A key challenge for deep learning models is to work with millions of categorical classes or tokens. We propose a novel formulation of memory shared embedding, where memory is shared in proportion to the overlap in semantic information. We demonstrate a significant reduction in the memory footprint while maintaining performance.
  arXiv  Detail & Related papers  (2021-02-24T19:55:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.