How LoRA Remembers? A Parametric Memory Law for LLM Finetuning

• URL: http://arxiv.org/abs/2605.30260v1
• Date: Thu, 28 May 2026 17:22:24 GMT
• Title: How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
• Abstract summary: We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length.<n>At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for recall.<n>MemFT is a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens.
• Score: 29.141706047011862
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.

Related papers

• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents [97.19033877543522]
  We present a controlled harness evaluation of memory substrates for memory-augmented agents.<n>We instrument 26 performance and efficiency metrics under a unified harness.<n>These findings motivate substrate routing as a necessary component of adaptive agent memory systems.
  arXiv  Detail & Related papers  (2026-08-15T03:29:48Z)
• Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory [50.857546269660276]
  We introduce SeqMem-Eval, a diagnostic evaluation framework for sequentially evolving large language models (LLMs) memory.<n>Rather than focusing only on final performance, SeqMem-Eval evaluates how memory states evolve, generalize, consolidate experience, and retain useful information during sequential inference.
  arXiv  Detail & Related papers  (2026-05-14T20:15:22Z)
• Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation [1.2559585990041289]
  LARS (Low-memory Activation-Rank Subspace) is a novel adaptation framework that decouples memory consumption from sequence length.<n>We show that LARS reduces the memory footprint by an average of 33.54% on GPUs and 51.95% on CPUs in comparison to LoRA across reasoning, understanding and long-context datasets.
  arXiv  Detail & Related papers  (2026-04-03T17:05:33Z)
• Adaptive Memory Admission Control for LLM Agents [9.04001220868675]
  We propose Adaptive Memory Admission Control (A-MAC), a framework that treats memory admission as a structured decision problem.<n>A-MAC decomposes memory value into five complementary and interpretable factors: future utility, factual confidence, semantic novelty, temporal recency, and content type prior.<n>A-MAC achieves a superior precision-recall tradeoff, improving F1 to 0.583 while reducing latency by 31% compared to state-of-the-art LLM-native memory systems.
  arXiv  Detail & Related papers  (2026-03-04T19:32:02Z)
• NextMem: Towards Latent Factual Memory for LLM-based Agents [58.35585202907478]
  NextMem is a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory.<n>Extensive experiments demonstrate that NextMem achieves superior performance.
  arXiv  Detail & Related papers  (2026-02-26T14:35:27Z)
• Training Large Reasoning Models Efficiently via Progressive Thought Encoding [63.254758972725654]
  Large reasoning models (LRMs) excel on complex problems but face a critical barrier to efficiency.<n>We introduce Progressive Thought, a parameter-efficient fine-tuning method that enables LRMs to reason effectively under fixed-size caches.
  arXiv  Detail & Related papers  (2026-02-18T20:03:38Z)
• Memory-Efficient Fine-Tuning via Low-Rank Activation Compression [16.44044624606008]
  Low-Rank Activation Compression (LoRAct) is a memory-efficient fine-tuning approach.<n>LoRAct reduces activation memory by approximately 80% in comparison with the widely adopted LoRA method.
  arXiv  Detail & Related papers  (2025-09-27T19:48:32Z)
• MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation [24.943207005554246]
  We propose a memory-efficient training paradigm called Momentum Low-rank compression (MLorc)<n>The key idea of MLorc is to compress and reconstruct the momentum of matrix parameters during training to reduce memory consumption.
  arXiv  Detail & Related papers  (2025-06-02T17:21:10Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• AdaLomo: Low-memory Optimization with Adaptive Learning Rate [59.64965955386855]
  We introduce low-memory optimization with adaptive learning rate (AdaLomo) for large language models. AdaLomo results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.
  arXiv  Detail & Related papers  (2023-10-16T09:04:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.