Unveiling and Eliminating the Shortcut Learning for Locate-Then-Edit Knowledge Editing via Both Subject and Relation Awareness

• URL: http://arxiv.org/abs/2506.04042v1
• Date: Wed, 04 Jun 2025 15:06:46 GMT
• Title: Unveiling and Eliminating the Shortcut Learning for Locate-Then-Edit Knowledge Editing via Both Subject and Relation Awareness
• Authors: Xiyu Liu, Zhengxiao Liu, Naibin Gu, Zheng Lin, Ji Xiang, Weiping Wang,
• Abstract summary: Knowledge editing aims to alternate the target knowledge predicted by large language models.<n>We propose a novel two-stage optimization process that balances the learning of the subject feature and the relation feature.
• Score: 15.781679300397562
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge editing aims to alternate the target knowledge predicted by large language models while ensuring the least side effects on unrelated knowledge. An effective way to achieve knowledge editing is to identify pivotal parameters for predicting factual associations and modify them with an optimization process to update the predictions. However, these locate-then-edit methods are uncontrollable since they tend to modify most unrelated relations connected to the subject of target editing. We unveil that this failure of controllable editing is due to a shortcut learning issue during the optimization process. Specifically, we discover two crucial features that are the subject feature and the relation feature for models to learn during optimization, but the current optimization process tends to over-learning the subject feature while neglecting the relation feature. To eliminate this shortcut learning of the subject feature, we propose a novel two-stage optimization process that balances the learning of the subject feature and the relation feature. Experimental results demonstrate that our approach successfully prevents knowledge editing from shortcut learning and achieves the optimal overall performance, contributing to controllable knowledge editing.

Related papers

• KEDAS: Knowledge Editing Alignment with Diverse Augmentation and Self-adaptive Inference [8.634349480743873]
  We propose Knowledge Editing alignment with Diverse Augmentation and Self-adaptive inference (KEDAS) to better align large language models with knowledge editing.<n>In experiments, KEDAS secures the highest overall performance scores in 35 out of 36 cases across four datasets.
  arXiv  Detail & Related papers  (2025-08-02T10:25:36Z)
• Retention analysis of edited knowledge after fine-tuning [5.440397659472036]
  Large language models (LLMs) store vast amounts of knowledge, which often requires updates to correct factual errors, incorporate newly acquired information, or adapt model behavior.<n>Model editing methods have emerged as efficient solutions for such updates, offering localized and precise knowledge modification at significantly lower computational cost than continual training.<n>However, the effect of fine-tuning on previously edited knowledge remains poorly understood.
  arXiv  Detail & Related papers  (2025-07-14T15:51:19Z)
• InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing [77.47790551485721]
  In-context learning is a promising editing method by comprehending edit information through context encoding.<n>This method is constrained by the limited context window of large language models.<n>We propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts.
  arXiv  Detail & Related papers  (2025-05-28T09:20:18Z)
• K-Edit: Language Model Editing with Contextual Knowledge Awareness [71.73747181407323]
  Knowledge-based model editing enables precise modifications to the weights of large language models.<n>We present K-Edit, an effective approach to generating contextually consistent knowledge edits.
  arXiv  Detail & Related papers  (2025-02-15T01:35:13Z)
• Related Knowledge Perturbation Matters: Rethinking Multiple Pieces of Knowledge Editing in Same-Subject [49.559994791305535]
  Current state-of-the-art editing methods struggle when tasked with editing multiple related knowledge pieces for the same subject.<n>We introduce the $textS2textRKE$(Same-Subject Related Knowledge Editing) benchmark.<n>Our experiments reveal that only mainstream locate-then-edit methods, such as ROME and MEMIT, exhibit "related knowledge perturbation"
  arXiv  Detail & Related papers  (2025-02-08T04:47:17Z)
• Uncovering Overfitting in Large Language Model Editing [35.55260822503773]
  We identify and investigate the phenomenon of Editing Overfit, where edited models assign disproportionately high probabilities to the edit target. We propose a new plug-and-play strategy called Learn to Inference (LTI), which introduce a Multi-stage Inference Constraint module to guide the edited models in recalling new knowledge.
  arXiv  Detail & Related papers  (2024-10-10T11:09:00Z)
• EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundaries [69.72012539060731]
  We introduce a theoretical framework for efficient knowledge editing (KE) in large language models (LLMs) We propose a novel task of event-based knowledge editing that pairs facts with event descriptions. We empirically demonstrate the superiority of event-based editing over the existing setting on resolving uncertainty in edited models.
  arXiv  Detail & Related papers  (2024-02-17T16:34:50Z)
• Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks [36.292901021210575]
  We introduce a novel reasoning-based benchmark -- ReCoE (Reasoning-based Counterfactual Editing dataset) We conduct a thorough analysis of existing knowledge editing techniques, including input augmentation, finetuning, and locate-and-edit. All model editing methods show notably low performance on this dataset, especially in certain reasoning schemes.
  arXiv  Detail & Related papers  (2024-01-31T04:12:59Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)
• Editing Large Language Models: Problems, Methods, and Opportunities [51.903537096207]
  This paper embarks on a deep exploration of the problems, methods, and opportunities related to model editing for LLMs. We provide an exhaustive overview of the task definition and challenges associated with model editing, along with an in-depth empirical analysis of the most progressive methods currently at our disposal. Our objective is to provide valuable insights into the effectiveness and feasibility of each editing technique, thereby assisting the community in making informed decisions on the selection of the most appropriate method for a specific task or context.
  arXiv  Detail & Related papers  (2023-05-22T16:00:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.