Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

• URL: http://arxiv.org/abs/2606.22627v1
• Date: Sun, 21 Jun 2026 18:29:40 GMT
• Title: Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs
• Abstract summary: Knowledge editing aims to efficiently update factual information in Large Language Models (LLMs) without full retraining.<n>Existing methods still suffer from performance degradation in batch knowledge editing.<n>We propose Orthogonal Representation Editing (ORE), which performs edits in the hidden representation space of LLMs.
• Score: 38.268937785139855
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge editing aims to efficiently update factual information in Large Language Models (LLMs) without full retraining. However, existing methods still suffer from performance degradation in batch knowledge editing. We identify that semantic representation entanglement, such as overlapping concepts and shared syntactic patterns, accumulates interference in the representation space and reduces editing precision. To bridge this gap, in this paper, we propose Orthogonal Representation Editing (ORE), which performs edits in the hidden representation space of LLMs by constructing a general semantic subspace and enforcing orthogonal constraints on edit vectors, effectively decoupling semantic entanglement. Furthermore, we introduce a gated non-linear representation head to enable adaptive learning of editing locations and precise control over knowledge injection. Extensive experiments show that ORE outperforms existing methods and achieves superior performance in cross-lingual knowledge editing scenarios. We release our code at https://github.com/YVVH/ORE.

Related papers

• DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editing [1.6830191160943109]
  Lifelong editing is a challenging task, prone to disrupting previously learned concepts.<n>Current methods algorithmically control edits via optimization rather than structurally separating knowledge.<n>We introduce Dynamic Subspace Concept Alignment (DSCA) which by design mitigates this limitation.<n>Our method achieves 98 percent single edit success, remains over 95 percent after 1000 sequential edits, hallucination by 3 to 5 percent, and the best backward transfer (BWT) scores on continual instruction tuning benchmarks.
  arXiv  Detail & Related papers  (2026-04-09T08:25:54Z)
• Consistency-Aware Editing for Entity-level Unlearning in Language Models [53.522931419965424]
  We introduce a novel consistency-aware editing (CAE) framework for entity-level unlearning.<n>CAE aggregates a diverse set of prompts related to a target entity, including its attributes, relations, and adversarial paraphrases.<n>It then jointly learns a low-rank update guided by a consistency regularizer that aligns the editing directions across prompts.
  arXiv  Detail & Related papers  (2025-12-19T15:18:07Z)
• Representation Interventions Enable Lifelong Unstructured Knowledge Control [54.86207134539453]
  Large language models (LLMs) often produce incorrect or outdated content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge.<n>We introduce RILKE, a robust and scalable method that treats knowledge control as interventions within the model's representation space.<n>During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference.<n>In inference, a query-adaptive router selects the appropriate module to guide the model's generation.
  arXiv  Detail & Related papers  (2025-11-25T22:15:00Z)
• Latent Knowledge Scalpel: Precise and Massive Knowledge Editing for Large Language Models [3.834827405473377]
  Large Language Models (LLMs) often retain inaccurate or outdated information from pre-training, leading to incorrect predictions or biased outputs during inference.<n>We introduce the Latent Knowledge Scalpel (LKS), an LLM editor that manipulates the latent knowledge of specific entities via a lightweight hypernetwork to enable precise and large-scale editing.<n> Experiments conducted on Llama-2 and Mistral show even with the number of simultaneous edits reaching 10,000, LKS effectively performs knowledge editing while preserving the general abilities of the edited LLMs.
  arXiv  Detail & Related papers  (2025-08-01T03:51:43Z)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs [76.28901550926021]
  Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, while preserving the core capabilities of the pre-trained model.<n>MeMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.
  arXiv  Detail & Related papers  (2025-06-09T16:16:42Z)
• InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing [86.17245523439514]
  In-context learning is a promising editing method by comprehending edit information through context encoding.<n>This method is constrained by the limited context window of large language models.<n>We propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts.
  arXiv  Detail & Related papers  (2025-05-28T09:20:18Z)
• REACT: Representation Extraction And Controllable Tuning to Overcome Overfitting in LLM Knowledge Editing [42.89229070245538]
  We introduce REACT, a framework for precise and controllable knowledge editing.<n>In the initial phase, we utilize tailored stimuli to extract latent factual representations.<n>In the second phase, we apply controllable perturbations to hidden states using the obtained vector with a magnitude scalar.
  arXiv  Detail & Related papers  (2025-05-25T01:57:06Z)
• SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering [17.20346072074533]
  Recent model editing is a promising technique for efficiently updating a small amount of knowledge of large language models.<n>We propose a detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching.<n>We demonstrate the overall state-of-the-art (SOTA) performance of SWEA$oplus$OS on the CounterFact and zsRE datasets.
  arXiv  Detail & Related papers  (2024-01-31T13:08:45Z)
• Editing Arbitrary Propositions in LLMs without Subject Labels [88.67755930096966]
  We introduce a simple and fast localization method called Gradient Tracing (GT) GT allows editing arbitrary propositions instead of just binary ones, and does so without the need for subject labels. We show that our method -- without access to subject labels -- performs close to state-of-the-art L&E methods which has access subject labels.
  arXiv  Detail & Related papers  (2024-01-15T08:08:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.