Reversible Lifelong Model Editing via Semantic Routing-Based LoRA

• URL: http://arxiv.org/abs/2603.11239v2
• Date: Thu, 19 Mar 2026 06:13:33 GMT
• Title: Reversible Lifelong Model Editing via Semantic Routing-Based LoRA
• Abstract summary: We propose SoLA, a Semantic routing-based LoRA framework for lifelong model editing.<n>Each edit is encapsulated as an independent LoRA module, which is frozen after training and mapped to input by semantic routing.<n>SoLA supports precise revocation of specific edits by removing key from semantic routing, which restores model's original behavior.
• Score: 28.767885121925406
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The dynamic evolution of real-world necessitates model editing within Large Language Models. While existing methods explore modular isolation or parameter-efficient strategies, they still suffer from semantic drift or knowledge forgetting due to continual updating. To address these challenges, we propose SoLA, a Semantic routing-based LoRA framework for lifelong model editing. In SoLA, each edit is encapsulated as an independent LoRA module, which is frozen after training and mapped to input by semantic routing, allowing dynamic activation of LoRA modules via semantic matching. This mechanism avoids semantic drift caused by cluster updating and mitigates catastrophic forgetting from parameter sharing. More importantly, SoLA supports precise revocation of specific edits by removing key from semantic routing, which restores model's original behavior. To our knowledge, this reversible rollback editing capability is the first to be achieved in existing literature. Furthermore, SoLA integrates decision-making process into edited layer, eliminating the need for auxiliary routing networks and enabling end-to-end decision-making process. Extensive experiments demonstrate that SoLA effectively learns and retains edited knowledge, achieving accurate, efficient, and reversible lifelong model editing.

Related papers

• JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models [78.56572636138429]
  We propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating.<n>We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
  arXiv  Detail & Related papers  (2026-04-17T15:38:37Z)
• Consistency-Aware Editing for Entity-level Unlearning in Language Models [53.522931419965424]
  We introduce a novel consistency-aware editing (CAE) framework for entity-level unlearning.<n>CAE aggregates a diverse set of prompts related to a target entity, including its attributes, relations, and adversarial paraphrases.<n>It then jointly learns a low-rank update guided by a consistency regularizer that aligns the editing directions across prompts.
  arXiv  Detail & Related papers  (2025-12-19T15:18:07Z)
• Recover-to-Forget: Gradient Reconstruction from LoRA for Efficient LLM Unlearning [17.898277374771254]
  We introduce Recover-to-Forget (R2F), a novel framework for efficient unlearning in large foundation models.<n>R2F reconstructs full-model gradient directions from low-rank LoRA adapter updates.<n>We show that R2F offers a scalable and lightweight alternative for unlearning in pretrained LLMs without requiring full retraining or access to internal parameters.
  arXiv  Detail & Related papers  (2025-12-08T10:10:12Z)
• Representation Interventions Enable Lifelong Unstructured Knowledge Control [54.86207134539453]
  Large language models (LLMs) often produce incorrect or outdated content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge.<n>We introduce RILKE, a robust and scalable method that treats knowledge control as interventions within the model's representation space.<n>During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference.<n>In inference, a query-adaptive router selects the appropriate module to guide the model's generation.
  arXiv  Detail & Related papers  (2025-11-25T22:15:00Z)
• UnGuide: Learning to Forget with LoRA-Guided Diffusion Models [6.860380947025009]
  Recent advances in large-scale text-to-image diffusion models have heightened concerns about their misuse.<n>This underscores the need for effective machine unlearning, i.e., removing specific knowledge or concepts from pretrained models.<n>We present UnGuide, a novel approach which incorporates Un-Free Guidance (CFG) to exert precise control over the unlearning process.
  arXiv  Detail & Related papers  (2025-08-07T18:12:03Z)
• AutoLoRA: Automatic LoRA Retrieval and Fine-Grained Gated Fusion for Text-to-Image Generation [32.46570968627392]
  Low-rank adaptation (LoRA) have demonstrated efficacy in enabling model customization with minimal parameter overhead.<n>We introduce a novel framework that enables semantic-driven LoRA retrieval and dynamic aggregation.<n>Our approach achieves significant improvement in image generation perfermance.
  arXiv  Detail & Related papers  (2025-08-04T06:36:00Z)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs [76.28901550926021]
  Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, while preserving the core capabilities of the pre-trained model.<n>MeMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.
  arXiv  Detail & Related papers  (2025-06-09T16:16:42Z)
• InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing [86.17245523439514]
  In-context learning is a promising editing method by comprehending edit information through context encoding.<n>This method is constrained by the limited context window of large language models.<n>We propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts.
  arXiv  Detail & Related papers  (2025-05-28T09:20:18Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors.<n>Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update.<n>We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering [17.20346072074533]
  Recent model editing is a promising technique for efficiently updating a small amount of knowledge of large language models.<n>We propose a detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching.<n>We demonstrate the overall state-of-the-art (SOTA) performance of SWEA$oplus$OS on the CounterFact and zsRE datasets.
  arXiv  Detail & Related papers  (2024-01-31T13:08:45Z)
• Emptying the Ocean with a Spoon: Should We Edit Models? [8.545919917068273]
  We call into question the recently popularized method of direct model editing as a means of correcting factual errors in LLM generations. We contrast model editing with three similar but distinct approaches that pursue better defined objectives.
  arXiv  Detail & Related papers  (2023-10-18T13:38:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.