Energy-Regularized Sequential Model Editing on Hyperspheres

• URL: http://arxiv.org/abs/2510.01172v1
• Date: Wed, 01 Oct 2025 17:55:43 GMT
• Title: Energy-Regularized Sequential Model Editing on Hyperspheres
• Authors: Qingyuan Liu, Jia-Chen Gu, Yunzhi Yao, Hong Wang, Nanyun Peng,
• Abstract summary: Large language models (LLMs) require constant updates to remain aligned with evolving real-world knowledge.<n> sequential editing often destabilizes representations and induces catastrophic forgetting.<n>We propose SPHERE (Sparse Projection for Hyperspherical Energy-Regularized Editing), an HE-driven regularization strategy that stabilizes neuron weight distributions.
• Score: 59.47007547581175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) require constant updates to remain aligned with evolving real-world knowledge. Model editing offers a lightweight alternative to retraining, but sequential editing often destabilizes representations and induces catastrophic forgetting. In this work, we seek to better understand and mitigate performance degradation caused by sequential editing. We hypothesize that hyperspherical uniformity, a property that maintains uniform distribution of neuron weights on a hypersphere, helps the model remain stable, retain prior knowledge, while still accommodate new updates. We use Hyperspherical Energy (HE) to quantify neuron uniformity during editing, and examine its correlation with editing performance. Empirical studies across widely used editing methods reveals a strong correlation between HE dynamics and editing performance, with editing failures consistently coinciding with high HE fluctuations. We further theoretically prove that HE dynamics impose a lower bound on the degradation of pretrained knowledge, highlighting why HE stability is crucial for knowledge retention. Motivated by these insights, we propose SPHERE (Sparse Projection for Hyperspherical Energy-Regularized Editing), an HE-driven regularization strategy that stabilizes neuron weight distributions, ultimately preserving prior knowledge while enabling reliable sequential updates. Specifically, SPHERE identifies a sparse space complementary to the principal hyperspherical directions of the pretrained weight matrices and projects new knowledge onto it, attenuating perturbations on the principal directions. Extensive experiments on LLaMA3 (8B) and Qwen2.5 (7B) show that SPHERE outperforms the best baseline in editing capability by an average of 16.41%, while most faithfully preserving general model performance, thereby offering a principled path toward reliable large-scale knowledge editing.

Related papers

• Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation [58.3773038915023]
  Continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge.<n>Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge.<n>We address this problem using a projected first-order method compatible with standard deep-dots used in vision-language models.
  arXiv  Detail & Related papers  (2026-01-31T13:27:02Z)
• Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse [44.49646322759214]
  We show that a model's general abilities are closely associated with dominant singular directions of pretrained weight matrices.<n>We propose REVIVE, a plug-and-play framework that stabilizes sequential editing by explicitly preserving the dominant singular subspace.
  arXiv  Detail & Related papers  (2026-01-16T07:18:14Z)
• Massive Editing for Large Language Models Based on Dynamic Weight Generation [51.34392079812964]
  This paper proposes a Massive editing approach for Large Language Models (LLMs) based on dynamic weight Generation (MeG)<n>Our MeG can significantly improve the performance of large-scale knowledge editing in terms of Reliability, Generality, and Locality metrics.
  arXiv  Detail & Related papers  (2025-12-16T13:32:55Z)
• STABLE: Gated Continual Learning for Large Language Models [0.0]
  STABLE is a gated continual self editing framework that constrains forgetting during sequential updates.<n>Each candidate edit is evaluated against a stability budget using one of three metrics.<n>Experiments on the Qwen-2.5-7B model show that gating effectively mitigates forgetting while preserving adaptability.
  arXiv  Detail & Related papers  (2025-10-17T16:14:05Z)
• EvoEdit: Evolving Null-space Alignment for Robust and Efficient Knowledge Editing [19.834477925624658]
  Large language models (LLMs) require continual updates to rectify outdated or erroneous knowledge.<n>Existing approaches are mainly based on a locate-then-edit framework.<n>We introduce EvoEdit, a novel editing strategy that mitigates catastrophic interference through sequential null-space alignment.
  arXiv  Detail & Related papers  (2025-10-11T21:36:14Z)
• Retention analysis of edited knowledge after fine-tuning [5.440397659472036]
  Large language models (LLMs) store vast amounts of knowledge, which often requires updates to correct factual errors, incorporate newly acquired information, or adapt model behavior.<n>Model editing methods have emerged as efficient solutions for such updates, offering localized and precise knowledge modification at significantly lower computational cost than continual training.<n>However, the effect of fine-tuning on previously edited knowledge remains poorly understood.
  arXiv  Detail & Related papers  (2025-07-14T15:51:19Z)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs [76.28901550926021]
  Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, while preserving the core capabilities of the pre-trained model.<n>MeMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.
  arXiv  Detail & Related papers  (2025-06-09T16:16:42Z)
• Model Editing with Graph-Based External Memory [12.694485038895813]
  We propose a novel framework that leverages hyperbolic geometry and graph neural networks for precise and stable model edits.<n> Experiments on CounterFact, CounterFact+, and MQuAKE with GPT-J and GPT2-XL demonstrate that HYPE significantly enhances edit stability, factual accuracy, and multi-hop reasoning.
  arXiv  Detail & Related papers  (2025-05-23T19:57:51Z)
• LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editing [27.918524905286475]
  Current locate-then-edit approaches exhibit a progressive performance decline during sequential editing.<n>textbfLyapLock is proposed to decompose the long-term constrained programming into tractable stepwise subproblems for efficient solving.<n> Experimental results show that our framework scales sequential editing capacity to over 10,000 edits while stabilizing general capabilities and boosting average editing efficacy by 11.89% over SOTA baselines.
  arXiv  Detail & Related papers  (2025-05-21T16:16:33Z)
• Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue [122.20016030723043]
  We evaluate the side effects of model editing on large language models (LLMs) Our analysis reveals that the side effects are caused by model editing altering the original model weights excessively. To mitigate this, a method named RECT is proposed to regularize the edit update weights.
  arXiv  Detail & Related papers  (2024-01-09T18:03:15Z)
• Memory-Based Model Editing at Scale [102.28475739907498]
  Existing model editors struggle to accurately model an edit's intended scope. We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC) SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
  arXiv  Detail & Related papers  (2022-06-13T23:40:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.