MemSFT: Mitigating Alignment Tax with an External Parametric Memory

• URL: http://arxiv.org/abs/2607.25614v1
• Date: Tue, 28 Jul 2026 11:45:34 GMT
• Title: MemSFT: Mitigating Alignment Tax with an External Parametric Memory
• Abstract summary: Adapting Large Language Models to specialized domains often incurs an alignment tax.<n>We propose MemSFT, which mitigates the alignment tax by decoupling domain specialization from backbone parameter updates.<n>We show that MemSFT consistently improves domain performance with negligible degradation in general performance, whereas full SFT suffers severe forgetting on general tasks.
• Score: 39.4211715596737
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adapting Large Language Models (LLMs) to specialized domains often incurs an alignment tax, as fine-tuning on domain-specific tasks can cause catastrophic forgetting and substantially degrade performance on general tasks. We propose MemSFT, which mitigates the alignment tax by decoupling domain specialization from backbone parameter updates through a plug-and-play parametric memory. The memory is trained to imitate the behavior of a non-parametric retriever operating over domain data, thereby memorizing knowledge and patterns that would otherwise be accessed through retrieval. Once trained on a specific domain, the memory can be reused across LLMs of different sizes. During generation, a learned router dynamically fuses the output distributions of the memory and backbone at each decoding step, allowing domain expertise to be invoked selectively. Across biology, geoscience, and law, evaluations with models ranging from Qwen3-8B to Qwen3-235B-A22B show that MemSFT consistently improves domain performance with negligible degradation in general performance, whereas full SFT suffers severe forgetting on general tasks. Overall, our results demonstrate a practical path to decoupling general model capabilities from domain-specific knowledge at the parameter level, thereby equipping LLMs with new specialized capabilities without compromising their general capabilities.

Related papers

• Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner [55.80248281761638]
  multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning.<n>We propose DG-Mem, a dual-grained agentic memory framework that augments a frozen MLLM with a non- externally stored memory built once from training-time rollouts and consulted read-only at test time.
  arXiv  Detail & Related papers  (2026-08-24T13:56:01Z)
• UniMem: Complementary Episodic-to-Parametric Memory for Boundary-Agnostic Task Streams [77.52140119830241]
  We propose UniMem, a self-routing framework for autonomous memory management.<n>UniMem uses learnable routing tokens as memory controllers, enabling adaptive coordination between complementary memory pathways.<n>Experiments on long-horizon streaming task sequences show that UniMem consistently outperforms baselines while maintaining execution fidelity.
  arXiv  Detail & Related papers  (2026-07-28T17:28:21Z)
• MILE: Mixture of Incremental LoRA Experts for Continual Semantic Segmentation across Domains and Modalities [19.243037567371875]
  Continual semantic segmentation requires models to adapt to new domains or modalities without sacrificing performance on previously learned tasks.<n>We introduce MILE, a modular and parameter-efficient framework for continual segmentation across both domains and modalities.
  arXiv  Detail & Related papers  (2026-05-05T09:27:31Z)
• Memory Transfer Learning: How Memories are Transferred Across Domains in Coding Agents [56.078070829341534]
  Memory-based self-evolution has emerged as a promising paradigm for coding agents.<n>We investigate textbfMemory Transfer Learning (MTL) by harnessing a unified memory pool from heterogeneous domains.
  arXiv  Detail & Related papers  (2026-04-15T15:50:29Z)
• MEMTS: Internalizing Domain Knowledge via Parameterized Memory for Retrieval-Free Domain Adaptation of Time Series Foundation Models [51.506429027626005]
  Memory for Time Series (MEMTS) is a lightweight and plug-and-play method for retrieval-free domain adaptation in time series forecasting.<n>Key component of MEMTS is a Knowledge Persistence Module (KPM), which internalizes domain-specific temporal dynamics.<n>This paradigm shift enables MEMTS to achieve accurate domain adaptation with constant-time inference and near-zero latency.
  arXiv  Detail & Related papers  (2026-02-14T14:00:06Z)
• Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models [46.59786740489401]
  This paper introduces Memory Decoder, a plug-and-play pretrained memory that enables efficient domain adaptation without changing the original model's parameters.<n> Experimental results demonstrate that Memory Decoder enables effective adaptation of various Qwen and Llama models to three distinct specialized domains: biomedicine, finance, and law, reducing perplexity by an average of 6.17 points.
  arXiv  Detail & Related papers  (2025-08-13T15:16:29Z)
• MemOS: A Memory OS for AI System [116.87568350346537]
  Large Language Models (LLMs) have become an essential infrastructure for Artificial General Intelligence (AGI)<n>Existing models mainly rely on static parameters and short-lived contextual states, limiting their ability to track user preferences or update knowledge over extended periods.<n>MemOS is a memory operating system that treats memory as a manageable system resource.
  arXiv  Detail & Related papers  (2025-07-04T17:21:46Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• TAIA: Large Language Models are Out-of-Distribution Data Learners [30.57872423927015]
  We propose an effective inference-time intervention method: Training All parameters but Inferring with only Attention (trainallInfAttn) trainallInfAttn achieves superior improvements compared to both the fully fine-tuned model and the base model in most scenarios. The high tolerance of trainallInfAttn to data mismatches makes it resistant to jailbreaking tuning and enhances specialized tasks using general data.
  arXiv  Detail & Related papers  (2024-05-30T15:57:19Z)
• Learning Transferable Parameters for Unsupervised Domain Adaptation [29.962241958947306]
  Untrivial domain adaptation (UDA) enables a learning machine to adapt from a labeled source domain to an unlabeled domain under the distribution shift. We propose Transferable Learning (TransPar) to reduce the side effect brought by domain-specific information in the learning process.
  arXiv  Detail & Related papers  (2021-08-13T09:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.