Distributed Multi-Layer Editing for Rule-Level Knowledge in Large Language Models

• URL: http://arxiv.org/abs/2604.08284v1
• Date: Thu, 09 Apr 2026 14:22:19 GMT
• Title: Distributed Multi-Layer Editing for Rule-Level Knowledge in Large Language Models
• Abstract summary: Large language models store not only isolated facts but also rules that support reasoning across symbolic expressions, natural language explanations, and concrete instances.<n>Most model editing methods are built for fact-level knowledge, assuming that a target edit can be achieved through a localized intervention.<n>We investigate this problem through a mechanistic study of rule-level knowledge editing.
• Score: 56.355176278601476
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models store not only isolated facts but also rules that support reasoning across symbolic expressions, natural language explanations, and concrete instances. Yet most model editing methods are built for fact-level knowledge, assuming that a target edit can be achieved through a localized intervention. This assumption does not hold for rule-level knowledge, where a single rule must remain consistent across multiple interdependent forms. We investigate this problem through a mechanistic study of rule-level knowledge editing. To support this study, we extend the RuleEdit benchmark from 80 to 200 manually verified rules spanning mathematics and physics. Fine-grained causal tracing reveals a form-specific organization of rule knowledge in transformer layers: formulas and descriptions are concentrated in earlier layers, while instances are more associated with middle layers. These results suggest that rule knowledge is not uniformly localized, and therefore cannot be reliably edited by a single-layer or contiguous-block intervention. Based on this insight, we propose Distributed Multi-Layer Editing (DMLE), which applies a shared early-layer update to formulas and descriptions and a separate middle-layer update to instances. While remaining competitive on standard editing metrics, DMLE achieves substantially stronger rule-level editing performance. On average, it improves instance portability and rule understanding by 13.91 and 50.19 percentage points, respectively, over the strongest baseline across GPT-J-6B, Qwen2.5-7B, Qwen2-7B, and LLaMA-3-8B. The code is available at https://github.com/Pepper66/DMLE.

Related papers

• Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models [33.61747923589224]
  Large language models (LLMs) excel at text understanding and generation, yet struggle to reliably understand and apply externally provided procedural rules at scale.<n>We introduce RuleWorld, a large-scale benchmark that reformulates rules as globally reusable abstract units rather than instance-specific facts.<n>We propose DynaRule, an end-to-end framework that injects the given rules into the KV cache and turns retrieval into an internal, learnable, step-wise process.
  arXiv  Detail & Related papers  (2026-08-24T03:22:45Z)
• Towards Localized and Disentangled Knowledge Editing for Multimodal Large Language Models [12.330598426331091]
  Localized and Disentangled Knowledge Editing achieves precise and generalized editing by localizing fact-specific model layers and disentangling target-relevant inputs from irrelevant ones.<n>Experiments demonstrate LDKE achieves superior performance in propagating edits to related contexts while maintaining high locality.
  arXiv  Detail & Related papers  (2026-05-28T12:06:39Z)
• HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning [16.114622048845895]
  HiEdit is a hierarchical reinforcement learning framework that adaptively identifies the most knowledge-relevant layers for each editing instance.<n> Experiments show that HiEdit boosts the performance of the competitive RLEdit by an average of 8.48% with perturbing only half of the layers per edit.
  arXiv  Detail & Related papers  (2026-04-13T09:08:57Z)
• MiLDEdit: Reasoning-Based Multi-Layer Design Document Editing [67.28337411397062]
  We introduce the Multi-Layer Document Editing Agent (MiLDEAgent)<n>MiLDEAgent is a reasoning-based framework that combines an RL-trained multimodal reasoner for layer-wise understanding with an image editor for targeted modifications.<n>MiLDEAgent achieves strong layer-aware reasoning and precise editing, significantly outperforming all open-source baselines and attaining performance comparable to closed-source models.
  arXiv  Detail & Related papers  (2026-01-08T04:38:07Z)
• AnyEdit: Edit Any Knowledge Encoded in Language Models [76.28789588247659]
  We propose AnyEdit, a new autoregressive editing paradigm for large language models (LLMs)<n>It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs.<n>It outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge.
  arXiv  Detail & Related papers  (2025-02-08T16:18:37Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors.<n>Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update.<n>We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• Editing Arbitrary Propositions in LLMs without Subject Labels [88.67755930096966]
  We introduce a simple and fast localization method called Gradient Tracing (GT) GT allows editing arbitrary propositions instead of just binary ones, and does so without the need for subject labels. We show that our method -- without access to subject labels -- performs close to state-of-the-art L&E methods which has access subject labels.
  arXiv  Detail & Related papers  (2024-01-15T08:08:24Z)
• Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models [68.03946716358335]
  We find that we can change how a fact is stored in a model by editing weights that are in a different location than where existing methods suggest that the fact is stored. This is surprising because we would expect that localizing facts to specific model parameters would tell us where to manipulate knowledge in models. Our results suggest, counterintuitively, that better mechanistic understanding of how pretrained language models work may not always translate to insights about how to best change their behavior.
  arXiv  Detail & Related papers  (2023-01-10T21:26:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.