Toward Ultra-Long-Horizon Sequential Model Editing

• URL: http://arxiv.org/abs/2602.02543v1
• Date: Fri, 30 Jan 2026 04:31:21 GMT
• Title: Toward Ultra-Long-Horizon Sequential Model Editing
• Abstract summary: We show that long sequences of edits trigger abrupt model collapse in large language models.<n>We propose Norm-Anchor Scaling NAS, a plug-and-play norm-constrained strategy.<n>NAS delays the collapse point of representative L&E algorithms by more than 4 times and yields a 72.2% average relative gain in editing performance.
• Score: 0.43748379918040853
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model editing has emerged as a practical approach for mitigating factual errors and outdated knowledge in large language models (LLMs). Among existing methods, the Locate-and-Edit (L&E) paradigm is the dominant framework: it locates MLP parameters implicated in expressing a target fact, and then performs a localized update to rewrite that fact. However, long sequences of edits often trigger abrupt model collapse in L&E beyond a critical point. We empirically identify a strong correlation between collapse and explosive growth of edited MLP weight norms, and formally prove that commonly used L&E update rules can induce exponential norm growth across sequential edits in the absence of explicit norm control. To address this issue, we propose Norm-Anchor Scaling NAS, a plug-and-play norm-constrained strategy. Across extensive experiments, NAS delays the collapse point of representative L&E algorithms by more than 4 times and yields a 72.2% average relative gain in editing performance, requiring only a single additional line of code and incurring negligible computational overhead.

Related papers

• CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes [8.494161214945963]
  We introduce CritICL, a novel inference-time framework that improves reasoning while maintaining high efficiency.<n>We utilize failure modes derived from weaker models and incorporate them into inference through critique-based in-context examples.<n>CRITICL consistently outperforms standard in-context learning and achieves performance competitive with or superior to test-time scaling methods.
  arXiv  Detail & Related papers  (2026-08-27T17:59:30Z)
• Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models [12.330598426331091]
  Localized and Disentangled Knowledge Editing achieves precise and generalized editing by localizing fact-specific model layers and disentangling target-relevant inputs from irrelevant ones.<n>Experiments demonstrate LDKE achieves superior performance in propagating edits to related contexts while maintaining high locality.
  arXiv  Detail & Related papers  (2026-05-28T12:06:39Z)
• AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution [61.593935260052795]
  Post-training viaSupervised Fine-Tuning (SFT) and Reinforcement Learning (RL) is crucial for enhancing reasoning in Multimodal Large Language Models (MLLMs)<n>Existing paradigms often reach a performance bottleneck due to the limitations of static data.<n>We propose Anchor Evolution (AnE), a new paradigm that integrates truth-anchored data curation and model evolution.
  arXiv  Detail & Related papers  (2026-05-25T08:26:34Z)
• More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing [53.41147779846785]
  Lifelong Model Editing aims to continuously update evolving facts in Large Language Models while preserving unrelated knowledge and general capabilities.<n>We find that recent editors resilient over long horizons share the same core strategy: Lifelong Normalization (LN), which normalizes value using running statistics.<n>LN causes immediate performance collapse, and we observe a counter-intuitive positive cumulative effect where early edits can promote the success of future edits.
  arXiv  Detail & Related papers  (2026-05-12T09:23:35Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation [17.125957722393327]
  Stream of Revision is a paradigm shift that elevates code generation from a monotonic stream to a dynamic, self-correcting trajectory.<n>We introduce specific action tokens that enable the model to seamlessly backtrack and edit its own history within a single forward pass.
  arXiv  Detail & Related papers  (2026-02-01T12:22:46Z)
• STABLE: Gated Continual Learning for Large Language Models [0.0]
  STABLE is a gated continual self editing framework that constrains forgetting during sequential updates.<n>Each candidate edit is evaluated against a stability budget using one of three metrics.<n>Experiments on the Qwen-2.5-7B model show that gating effectively mitigates forgetting while preserving adaptability.
  arXiv  Detail & Related papers  (2025-10-17T16:14:05Z)
• Fine-tuning Done Right in Model Editing [83.79661791576103]
  Fine-tuning, a foundational method for adapting large language models, has long been considered ineffective for model editing.<n>We restore fine-tuning to the standard breadth-first (i.e., epoch-based) pipeline with mini-batch optimization.<n>We derive LocFT-BF, a simple and effective localized editing method built on the restored fine-tuning framework.
  arXiv  Detail & Related papers  (2025-09-26T08:53:13Z)
• VL Norm: Rethink Loss Aggregation in RLVR [23.196933474967224]
  We propose a loss aggregation method tailored to the characteristic of dynamic generation lengths in Reinforcement Learning with Verifiable Rewards (RLVR)<n>By analyzing the effect of varying lengths on policy loss both theoretically and empirically, we reformulate the problem as finding a minimum-variance unbiased estimator.<n>Our proposed VL Norm not only provides an unbiased estimate of the true policy loss but also minimizes gradient variance in theory.
  arXiv  Detail & Related papers  (2025-09-09T09:52:34Z)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs [76.28901550926021]
  Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, while preserving the core capabilities of the pre-trained model.<n>MeMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.
  arXiv  Detail & Related papers  (2025-06-09T16:16:42Z)
• LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editing [28.870053452479443]
  Current locate-then-edit approaches exhibit a progressive performance decline during sequential editing.<n>textbfLyapLock is proposed to decompose the long-term constrained programming into tractable stepwise subproblems for efficient solving.<n> Experimental results show that our framework scales sequential editing capacity to over 10,000 edits while stabilizing general capabilities and boosting average editing efficacy by 11.89% over SOTA baselines.
  arXiv  Detail & Related papers  (2025-05-21T16:16:33Z)
• Training Deep Learning Models with Norm-Constrained LMOs [56.00317694850397]
  We propose a new family of algorithms that uses the linear minimization oracle (LMO) to adapt to the geometry of the problem.<n>We demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam.
  arXiv  Detail & Related papers  (2025-02-11T13:10:34Z)
• Pushing the Limits of Large Language Model Quantization via the Linearity Theorem [71.3332971315821]
  We present a "line theoremarity" establishing a direct relationship between the layer-wise $ell$ reconstruction error and the model perplexity increase due to quantization. This insight enables two novel applications: (1) a simple data-free LLM quantization method using Hadamard rotations and MSE-optimal grids, dubbed HIGGS, and (2) an optimal solution to the problem of finding non-uniform per-layer quantization levels.
  arXiv  Detail & Related papers  (2024-11-26T15:35:44Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors.<n>Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update.<n>We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
  Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive. We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
  arXiv  Detail & Related papers  (2024-02-15T01:50:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.