Improving Sparse Memory Finetuning

• URL: http://arxiv.org/abs/2604.05248v1
• Date: Mon, 06 Apr 2026 23:16:10 GMT
• Title: Improving Sparse Memory Finetuning
• Abstract summary: Large Language Models (LLMs) are typically static after training, yet real-world applications require continual adaptation to new knowledge without degrading existing capabilities.<n>Standard approaches to updating models, like full finetuning or parameter-efficient methods, face a fundamental trade-off: catastrophic forgetting.<n>Sparse Memory Finetuning (SMF) offers a promising alternative by localizing updates to a small subset of parameters in explicit memory layers.
• Score: 6.491055956716639
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are typically static after training, yet real-world applications require continual adaptation to new knowledge without degrading existing capabilities. Standard approaches to updating models, like full finetuning or parameter-efficient methods (e.g., LoRA), face a fundamental trade-off: catastrophic forgetting. They modify shared dense representations, causing interference across tasks. Sparse Memory Finetuning (SMF) offers a promising alternative by localizing updates to a small subset of parameters in explicit memory layers. In this work, we present an open-source pipeline to retrofit existing pretrained models (Qwen-2.5-0.5B) with sparse memory modules, enabling effective continual learning on consumer hardware. We extend prior work by introducing a theoretically grounded slot-selection mechanism based on Kullback-Leibler (KL) divergence, which prioritizes memory updates for informationally "surprising" tokens relative to a background distribution. Our experiments demonstrate that our retrofitted models can acquire new factual knowledge with minimal forgetting of held-out capabilities, validating the sparse update hypothesis in a practical setting.

Related papers

• Towards Evolving Context Parameterization for Large Language Models [13.314849988154707]
  PLUME is a training-free method that constructs a global update representation and activates memory evidence to form a local parameter view.<n>We show PLUME's effectiveness in sequential evolution settings, yielding relative improvements of 29.9% in average ROUGE-L Recall and 54.9% in LLM-as-a-Judge.
  arXiv  Detail & Related papers  (2026-09-12T21:55:33Z)
• Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning [57.237835833636154]
  We propose MetrIc Learning with Expandable Subspace (Miles) to harness the prior information within pre-trained knowledge.<n>Miles achieves state-of-the-art performance in various CIL settings.
  arXiv  Detail & Related papers  (2026-07-20T06:22:02Z)
• MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models [64.2820121997882]
  MIITA is a Memory-Induced Inference-Time Adaptation framework for supervised CL under constrained storage.<n>It stores supervised experiences as compact correction-direction prototypes with semantic anchors, and retrieves them at inference time using semantic and uncertainty-based cues.<n>Experiments show that MIITA consistently improves final performance and mitigates forgetting under fixed memory budgets.
  arXiv  Detail & Related papers  (2026-05-20T03:03:04Z)
• Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories [44.46300411842271]
  Locas is a Locally-Supported parametric memory that shares the design of FFN blocks in modern transformers.<n>We show that proper initialization of such low-rank sideway-FFN-style memories is essential for fast convergence, improved generalization, and catastrophic prevention.
  arXiv  Detail & Related papers  (2026-02-04T22:09:40Z)
• Continual Learning via Sparse Memory Finetuning [58.163704181154834]
  We investigate whether sparse parameter updates can enable learning without catastrophic forgetting.<n>By updating only the memory slots that are highly activated by a new piece of knowledge relative to usage on pretraining data, we reduce interference between new knowledge and the model's existing capabilities.<n>We find that sparse memory finetuning learns new knowledge while exhibiting substantially less forgetting.
  arXiv  Detail & Related papers  (2025-10-16T19:44:38Z)
• Adaptive Additive Parameter Updates of Vision Transformers for Few-Shot Continual Learning [0.0]
  Few-shot class incremental learning (FSCIL) addresses this by first training a model on a robust dataset of base classes and then incrementally adapting it in successive sessions.<n>This approach is prone to overfitting on the limited new data, which can compromise overall performance and exacerbate forgetting.<n>We propose a novel FSCIL framework that leverages a frozen Vision Transformer (ViT) backbone augmented with parameter-efficient additive updates.
  arXiv  Detail & Related papers  (2025-04-11T21:17:30Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach [87.8330887605381]
  We show how to adapt a pre-trained Vision Transformer to downstream recognition tasks with only a few learnable parameters. We synthesize a task-specific query with a learnable and lightweight module, which is independent of the pre-trained model. Our method achieves state-of-the-art performance under memory constraints, showcasing its applicability in real-world situations.
  arXiv  Detail & Related papers  (2024-07-09T15:45:04Z)
• Co(ve)rtex: ML Models as storage channels and their (mis-)applications [2.792027541710663]
  In machine learning systems, don't-care states and undefined behavior have been shown to be sources of significant vulnerabilities. We consider the ML model as a storage channel with a capacity that increases with over parameterization. We develop optimizations to improve the capacity in this case, including a novel ML-specific substitution based error correction protocol.
  arXiv  Detail & Related papers  (2023-07-17T19:57:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.