Pioneering Reliable Assessment in Text-to-Image Knowledge Editing: Leveraging a Fine-Grained Dataset and an Innovative Criterion

• URL: http://arxiv.org/abs/2409.17928v2
• Date: Sat, 26 Oct 2024 06:03:00 GMT
• Title: Pioneering Reliable Assessment in Text-to-Image Knowledge Editing: Leveraging a Fine-Grained Dataset and an Innovative Criterion
• Authors: Hengrui Gu, Kaixiong Zhou, Yili Wang, Ruobing Wang, Xin Wang,
• Abstract summary: Text-to-Image (T2I) diffusion models encode factual knowledge into their parameters. Knowledge editing techniques aim to update model knowledge in a targeted way. We design a T2I knowledge editing framework by comprehensively spanning on three phases. We introduce textbfMPE, a simple but effective approach for T2I knowledge editing.
• Score: 21.37254997228105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: During pre-training, the Text-to-Image (T2I) diffusion models encode factual knowledge into their parameters. These parameterized facts enable realistic image generation, but they may become obsolete over time, thereby misrepresenting the current state of the world. Knowledge editing techniques aim to update model knowledge in a targeted way. However, facing the dual challenges posed by inadequate editing datasets and unreliable evaluation criterion, the development of T2I knowledge editing encounter difficulties in effectively generalizing injected knowledge. In this work, we design a T2I knowledge editing framework by comprehensively spanning on three phases: First, we curate a dataset \textbf{CAKE}, comprising paraphrase and multi-object test, to enable more fine-grained assessment on knowledge generalization. Second, we propose a novel criterion, \textbf{adaptive CLIP threshold}, to effectively filter out false successful images under the current criterion and achieve reliable editing evaluation. Finally, we introduce \textbf{MPE}, a simple but effective approach for T2I knowledge editing. Instead of tuning parameters, MPE precisely recognizes and edits the outdated part of the conditioning text-prompt to accommodate the up-to-date knowledge. A straightforward implementation of MPE (Based on in-context learning) exhibits better overall performance than previous model editors. We hope these efforts can further promote faithful evaluation of T2I knowledge editing methods.

Related papers

• Retention analysis of edited knowledge after fine-tuning [5.440397659472036]
  Large language models (LLMs) store vast amounts of knowledge, which often requires updates to correct factual errors, incorporate newly acquired information, or adapt model behavior.<n>Model editing methods have emerged as efficient solutions for such updates, offering localized and precise knowledge modification at significantly lower computational cost than continual training.<n>However, the effect of fine-tuning on previously edited knowledge remains poorly understood.
  arXiv  Detail & Related papers  (2025-07-14T15:51:19Z)
• Model Merging for Knowledge Editing [53.799891745131724]
  Large Language Models (LLMs) require continuous updates to maintain accurate and current knowledge as the world evolves.<n>Existing knowledge editing approaches offer various solutions for knowledge updating, but they often struggle with sequential editing scenarios.<n>This paper proposes a two-stage framework combining robust supervised fine-tuning (R-SFT) with model merging for knowledge editing.
  arXiv  Detail & Related papers  (2025-06-14T07:42:39Z)
• IA-T2I: Internet-Augmented Text-to-Image Generation [13.765327654914199]
  Current text-to-image (T2I) generation models achieve promising results, but they fail on the scenarios where the knowledge implied in the text prompt is uncertain.<n>We propose an Internet-Augmented text-to-image generation (IA-T2I) framework to compel T2I models clear about such uncertain knowledge by providing them with reference images.
  arXiv  Detail & Related papers  (2025-05-21T17:31:49Z)
• Understanding the Limits of Lifelong Knowledge Editing in LLMs [59.12302872055081]
  We bridge research into lifelong knowledge editing to real-world edits at practically relevant scale. We first introduce WikiBigEdit; a large-scale benchmark of real-world Wikidata edits. In its first instance, it includes over 500K question-answer pairs for knowledge editing.
  arXiv  Detail & Related papers  (2025-03-07T18:45:42Z)
• Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoning [38.018263569983226]
  We provide an evaluation of ten model editing methods along four dimensions: reliability, generalization, locality, and portability. We then propose a straightforward method called Selective Contextual Reasoning (SCR) for knowledge updating.
  arXiv  Detail & Related papers  (2025-03-07T08:04:25Z)
• The Mirage of Model Editing: Revisiting Evaluation in the Wild [70.17413507444704]
  We introduce QAEdit, a new benchmark aligned with widely used question answering (QA) datasets, and WILD, a task-agnostic evaluation framework.<n>Our single editing experiments show that current editing methods perform substantially worse than previously reported.
  arXiv  Detail & Related papers  (2025-02-16T15:57:55Z)
• K-Edit: Language Model Editing with Contextual Knowledge Awareness [71.73747181407323]
  Knowledge-based model editing enables precise modifications to the weights of large language models. We present K-Edit, an effective approach to generating contextually consistent knowledge edits.
  arXiv  Detail & Related papers  (2025-02-15T01:35:13Z)
• ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models [11.830273909934688]
  Modern Text-to-Image (T2I) Diffusion models have revolutionized image editing by enabling the generation of high-quality images. We propose ReEdit, a modular and efficient end-to-end framework that captures edits in both text and image modalities. Our results demonstrate that ReEdit consistently outperforms contemporary approaches both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2024-11-06T15:19:24Z)
• Detecting Edited Knowledge in Language Models [5.260519479124422]
  Knowledge editing methods (KEs) can update language models' obsolete or inaccurate knowledge learned from pre-training. Knowing whether a generated output is based on edited knowledge or first-hand knowledge from pre-training can increase users' trust in generative models. We propose a novel task: detecting edited knowledge in language models.
  arXiv  Detail & Related papers  (2024-05-04T22:02:24Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• AKEW: Assessing Knowledge Editing in the Wild [79.96813982502952]
  AKEW (Assessing Knowledge Editing in the Wild) is a new practical benchmark for knowledge editing. It fully covers three editing settings of knowledge updates: structured facts, unstructured texts as facts, and extracted triplets. Through extensive experiments, we demonstrate the considerable gap between state-of-the-art knowledge-editing methods and practical scenarios.
  arXiv  Detail & Related papers  (2024-02-29T07:08:34Z)
• Learning to Edit: Aligning LLMs with Knowledge Editing [101.96620267293731]
  We propose a Learning to Edit (LTE) framework, focusing on teaching large language models to apply updated knowledge into input questions. LTE features a two-phase process: (i) the Alignment Phase, which fine-tunes LLMs on a meticulously curated parallel dataset to make reliable, in-scope edits. We demonstrate LTE's superiority in knowledge editing performance, robustness in both batch and sequential editing, minimal interference on general tasks, and rapid editing speeds.
  arXiv  Detail & Related papers  (2024-02-19T07:45:17Z)
• EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundaries [69.72012539060731]
  We introduce a theoretical framework for efficient knowledge editing (KE) in large language models (LLMs) We propose a novel task of event-based knowledge editing that pairs facts with event descriptions. We empirically demonstrate the superiority of event-based editing over the existing setting on resolving uncertainty in edited models.
  arXiv  Detail & Related papers  (2024-02-17T16:34:50Z)
• StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing [86.92711729969488]
  We exploit the amazing capacities of pretrained diffusion models for the editing of images. They either finetune the model, or invert the image in the latent space of the pretrained model. They suffer from two problems: Unsatisfying results for selected regions, and unexpected changes in nonselected regions.
  arXiv  Detail & Related papers  (2023-03-28T00:16:45Z)
• EditEval: An Instruction-Based Benchmark for Text Improvements [73.5918084416016]
  This work presents EditEval: An instruction-based, benchmark and evaluation suite for automatic evaluation of editing capabilities. We evaluate several pre-trained models, which shows that InstructGPT and PEER perform the best, but that most baselines fall below the supervised SOTA. Our analysis shows that commonly used metrics for editing tasks do not always correlate well, and that optimization for prompts with the highest performance does not necessarily entail the strongest robustness to different models.
  arXiv  Detail & Related papers  (2022-09-27T12:26:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.