Knowledge Editing on Black-box Large Language Models

• URL: http://arxiv.org/abs/2402.08631v2
• Date: Sat, 17 Feb 2024 16:06:10 GMT
• Title: Knowledge Editing on Black-box Large Language Models
• Authors: Xiaoshuai Song, Zhengyang Wang, Keqing He, Guanting Dong, Yutao Mou, Jinxu Zhao, Weiran Xu
• Abstract summary: Knowledge editing aims to efficiently and precisely modify the behavior of large language models (LLMs) to update specific knowledge. Current research primarily focuses on white-box LLMs editing, overlooking an important scenario: black-box LLMs editing. We introduce KE on black-box LLMs and then propose a comprehensive evaluation framework to overcome the limitations of existing evaluations. Experiments and analysis on two benchmarks demonstrate that postEdit outperforms all baselines and achieves strong generalization.
• Score: 37.17131278142237
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge editing (KE) aims to efficiently and precisely modify the behavior of large language models (LLMs) to update specific knowledge without negatively influencing other knowledge. Current research primarily focuses on white-box LLMs editing, overlooking an important scenario: black-box LLMs editing, where LLMs are accessed through interfaces and only textual output is available. In this paper, we first officially introduce KE on black-box LLMs and then propose a comprehensive evaluation framework to overcome the limitations of existing evaluations that are not applicable to black-box LLMs editing and lack comprehensiveness. To tackle privacy leaks of editing data and style over-editing in current methods, we introduce a novel postEdit framework, resolving privacy concerns through downstream post-processing and maintaining textual style consistency via fine-grained editing to original responses. Experiments and analysis on two benchmarks demonstrate that postEdit outperforms all baselines and achieves strong generalization, especially with huge improvements on style retention (average $+20.82\%\uparrow$).

Related papers

• Bridging the Editing Gap in LLMs: FineEdit for Precise and Targeted Text Modifications [9.795246551841586]
  Large Language Models (LLMs) have transformed natural language processing, yet they still struggle with direct text editing tasks. In this work, we introduce a dual approach to enhance LLM editing performance. First, we present InstrEditBench, a high-quality benchmark dataset comprising over 20,000 structured editing tasks. Second, we propose FineEdit, a specialized model trained on this curated benchmark.
  arXiv  Detail & Related papers  (2025-02-19T01:41:44Z)
• AnyEdit: Edit Any Knowledge Encoded in Language Models [69.30638272162267]
  We propose AnyEdit, a new autoregressive editing paradigm for large language models (LLMs) It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. It outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge.
  arXiv  Detail & Related papers  (2025-02-08T16:18:37Z)
• Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts [17.376346967267327]
  We propose LiveEdit, a LIfelong Vision language modEl Edit to bridge the gap between lifelong LLM editing and Vision LLM editing. A hard filtering mechanism is developed to utilize visual semantic knowledge, thereby eliminating visually irrelevant experts for input queries. To integrate visually relevant experts, we introduce a soft routing mechanism based on textual semantic relevance to achieve multi-expert fusion.
  arXiv  Detail & Related papers  (2024-11-23T03:19:40Z)
• AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models [65.93240009586351]
  Large language models (LLMs) often exhibit hallucinations due to incorrect or outdated knowledge. We introduce AlphaEdit, a novel solution that projects perturbation onto the null space of the preserved knowledge before applying it to the parameters. We theoretically prove that this projection ensures the output of post-edited LLMs remains unchanged when queried about the preserved knowledge.
  arXiv  Detail & Related papers  (2024-10-03T10:06:27Z)
• Editing Conceptual Knowledge for Large Language Models [65.38231526537476]
  This paper pioneers the investigation of editing conceptual knowledge for Large Language Models (LLMs) We construct a novel benchmark dataset ConceptEdit and establish a suite of new metrics for evaluation. experimental results reveal that, although existing editing methods can efficiently modify concept-level definition to some extent, they also have the potential to distort the related instantial knowledge.
  arXiv  Detail & Related papers  (2024-03-10T16:57:10Z)
• Knowledge Graph Enhanced Large Language Model Editing [37.6721061644483]
  Large language models (LLMs) are pivotal in advancing natural language processing (NLP) tasks. Existing editing methods struggle to track and incorporate changes in knowledge associated with edits. We propose a novel model editing method that leverages knowledge graphs for enhancing LLM editing, namely GLAME.
  arXiv  Detail & Related papers  (2024-02-21T07:52:26Z)
• Learning to Edit: Aligning LLMs with Knowledge Editing [101.96620267293731]
  We propose a Learning to Edit (LTE) framework, focusing on teaching large language models to apply updated knowledge into input questions. LTE features a two-phase process: (i) the Alignment Phase, which fine-tunes LLMs on a meticulously curated parallel dataset to make reliable, in-scope edits. We demonstrate LTE's superiority in knowledge editing performance, robustness in both batch and sequential editing, minimal interference on general tasks, and rapid editing speeds.
  arXiv  Detail & Related papers  (2024-02-19T07:45:17Z)
• Beyond the Chat: Executable and Verifiable Text-Editing with LLMs [87.84199761550634]
  Conversational interfaces powered by Large Language Models (LLMs) have recently become a popular way to obtain feedback during document editing. We present InkSync, an editing interface that suggests executable edits directly within the document being edited.
  arXiv  Detail & Related papers  (2023-09-27T00:56:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.