UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Large Language Models

• URL: http://arxiv.org/abs/2505.14679v1
• Date: Tue, 20 May 2025 17:59:04 GMT
• Title: UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Large Language Models
• Authors: Xiaojie Gu, Guangxu Chen, Jungang Li, Jia-Chen Gu, Xuming Hu, Kai Zhang,
• Abstract summary: Lifelong learning enables large language models to adapt to evolving information by continually updating their internal knowledge.<n>Model editing stands out as a promising solution for this goal, offering a focused and efficient way to revise a model's internal knowledge.<n>UltraEdit is training-, subject- and memory-free, making it well-suited for ultra-scalable, real-world lifelong model editing.
• Score: 18.23723680134397
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Lifelong learning enables large language models (LLMs) to adapt to evolving information by continually updating their internal knowledge. An ideal system should support efficient, wide-ranging updates while preserving existing capabilities and ensuring reliable deployment. Model editing stands out as a promising solution for this goal, offering a focused and efficient way to revise a model's internal knowledge. Although recent paradigms have made notable progress, they often struggle to meet the demands of practical lifelong adaptation at scale. To bridge this gap, we propose ULTRAEDIT-a fundamentally new editing solution that is training-, subject- and memory-free, making it particularly well-suited for ultra-scalable, real-world lifelong model editing. ULTRAEDIT performs editing through a self-contained process that relies solely on lightweight linear algebra operations to compute parameter shifts, enabling fast and consistent parameter modifications with minimal overhead. To improve scalability in lifelong settings, ULTRAEDIT employs a lifelong normalization strategy that continuously updates feature statistics across turns, allowing it to adapt to distributional shifts and maintain consistency over time. ULTRAEDIT achieves editing speeds over 7x faster than the previous state-of-the-art method-which was also the fastest known approach-while consuming less than 1/3 the VRAM, making it the only method currently capable of editing a 7B LLM on a 24GB consumer-grade GPU. Furthermore, we construct ULTRAEDITBENCH-the largest dataset in the field to date, with over 2M editing pairs-and demonstrate that our method supports up to 1M edits while maintaining high accuracy. Comprehensive experiments on four datasets and six models show that ULTRAEDIT consistently achieves superior performance across diverse model editing scenarios. Our code is available at: https://github.com/XiaojieGu/UltraEdit.

Related papers

• Model Merging for Knowledge Editing [53.799891745131724]
  Large Language Models (LLMs) require continuous updates to maintain accurate and current knowledge as the world evolves.<n>Existing knowledge editing approaches offer various solutions for knowledge updating, but they often struggle with sequential editing scenarios.<n>This paper proposes a two-stage framework combining robust supervised fine-tuning (R-SFT) with model merging for knowledge editing.
  arXiv  Detail & Related papers  (2025-06-14T07:42:39Z)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs [82.34547399693966]
  Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory.<n>MeMOIR confines each edit to a distinct subset of the memory parameters, minimizing interference among edits.
  arXiv  Detail & Related papers  (2025-06-09T16:16:42Z)
• LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editing [27.918524905286475]
  Current locate-then-edit approaches exhibit a progressive performance decline during sequential editing.<n>textbfLyapLock is proposed to decompose the long-term constrained programming into tractable stepwise subproblems for efficient solving.<n> Experimental results show that our framework scales sequential editing capacity to over 10,000 edits while stabilizing general capabilities and boosting average editing efficacy by 11.89% over SOTA baselines.
  arXiv  Detail & Related papers  (2025-05-21T16:16:33Z)
• Reinforced Lifelong Editing for Language Models [12.101856766731574]
  Large language models (LLMs) acquire information from pre-training corpora, but their stored knowledge can become inaccurate or outdated over time.<n>Model editing addresses this challenge by modifying model parameters without retraining, and prevalent approaches leverage hypernetworks to generate these parameter updates.<n>We propose RLEdit, an RL-based editing method that captures changes at the full knowledge sequence level and generates appropriate parameter updates.
  arXiv  Detail & Related papers  (2025-02-09T03:37:06Z)
• O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing [0.0]
  Large language models (LLMs) acquire knowledge during pre-training, but over time, this knowledge may become incorrect or outdated, necessitating updates after training. We propose Orthogonal Subspace Editing, O-Edit. This algorithmizes the direction of each knowledge update, minimizing interference between successive updates and reducing the impact of new updates on unrelated knowledge. It can perform thousands of edits on mainstream LLMs, achieving an average performance improvement that is 4.2 times better than existing methods while effectively preserving the model's performance on downstream tasks, all with minimal additional parameter overhead.
  arXiv  Detail & Related papers  (2024-10-15T10:16:45Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors.<n>Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update.<n>We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models [30.831866499812925]
  Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts. We introduce LEMoE, an advanced Mixture of Experts (MoE) adaptor for lifelong model editing.
  arXiv  Detail & Related papers  (2024-06-28T16:17:41Z)
• Consecutive Batch Model Editing with HooK Layers [59.673084839708224]
  CoachHooK is a model editing method that simultaneously supports sequential and batch editing. It is memory-friendly as it only needs a small amount of it to store several hook layers whose size remains unchanged over time.
  arXiv  Detail & Related papers  (2024-03-08T14:07:44Z)
• Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors [53.819805242367345]
  We propose GRACE, a lifelong model editing method, which implements spot-fixes on streaming errors of a deployed model. GRACE writes new mappings into a pre-trained model's latent space, creating a discrete, local codebook of edits without altering model weights. Our experiments on T5, BERT, and GPT models show GRACE's state-of-the-art performance in making and retaining edits, while generalizing to unseen inputs.
  arXiv  Detail & Related papers  (2022-11-20T17:18:22Z)
• Memory-Based Model Editing at Scale [102.28475739907498]
  Existing model editors struggle to accurately model an edit's intended scope. We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC) SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
  arXiv  Detail & Related papers  (2022-06-13T23:40:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.