GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing

• URL: http://arxiv.org/abs/2505.11493v1
• Date: Fri, 16 May 2025 17:55:54 GMT
• Title: GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing
• Authors: Yusu Qian, Jiasen Lu, Tsu-Jui Fu, Xinze Wang, Chen Chen, Yinfei Yang, Wenze Hu, Zhe Gan,
• Abstract summary: We introduce a new benchmark designed to evaluate text-guided image editing models.<n>The benchmark includes over 1000 high-quality editing examples across 20 diverse content categories.<n>We conduct a large-scale study comparing GPT-Image-1 against several state-of-the-art editing models.
• Score: 60.66800567924348
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Editing images using natural language instructions has become a natural and expressive way to modify visual content; yet, evaluating the performance of such models remains challenging. Existing evaluation approaches often rely on image-text similarity metrics like CLIP, which lack precision. In this work, we introduce a new benchmark designed to evaluate text-guided image editing models in a more grounded manner, along two critical dimensions: (i) functional correctness, assessed via automatically generated multiple-choice questions that verify whether the intended change was successfully applied; and (ii) image content preservation, which ensures that non-targeted regions of the image remain visually consistent using an object-aware masking technique and preservation scoring. The benchmark includes over 1000 high-quality editing examples across 20 diverse content categories, each annotated with detailed editing instructions, evaluation questions, and spatial object masks. We conduct a large-scale study comparing GPT-Image-1, the latest flagship in the text-guided image editing space, against several state-of-the-art editing models, and validate our automatic metrics against human ratings. Results show that GPT-Image-1 leads in instruction-following accuracy, but often over-modifies irrelevant image regions, highlighting a key trade-off in the current model behavior. GIE-Bench provides a scalable, reproducible framework for advancing more accurate evaluation of text-guided image editing.

Related papers

• LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs [76.57152007140475]
  We introduce EBench-18K, the first large-scale image Editing Benchmark including 18K edited images with fine-grained human preference annotations.<n>EBench-18K includes 1,080 source images with corresponding editing prompts across 21 tasks, 18K+ edited images produced by 17 state-of-the-art TIE models, 55K+ mean opinion scores (MOSs) assessed from three evaluation dimensions, and 18K+ question-answering (QA) pairs.<n>Then, we propose LMM4Edit, a LMM-based metric for evaluating image Editing models from perceptual quality, editing alignment, attribute preservation
  arXiv  Detail & Related papers  (2025-07-22T03:11:07Z)
• EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits [22.762414256693265]
  We introduce EditInspector, a novel benchmark for evaluation of text-guided image edits.<n>We leverage EditInspector to evaluate the performance of state-of-the-art (SoTA) vision and language models in assessing edits.<n>Our findings indicate that current models struggle to evaluate edits comprehensively and frequently hallucinate when describing the changes.
  arXiv  Detail & Related papers  (2025-06-11T17:58:25Z)
• DCEdit: Dual-Level Controlled Image Editing via Precisely Localized Semantics [71.78350994830885]
  We present a novel approach to improving text-guided image editing using diffusion-based models.<n>Our method uses visual and textual self-attention to enhance the cross-attention map, which can serve as a regional cues to improve editing performance.<n>To fully compare our methods with other DiT-based approaches, we construct the RW-800 benchmark, featuring high resolution images, long descriptive texts, real-world images, and a new text editing task.
  arXiv  Detail & Related papers  (2025-03-21T02:14:03Z)
• IE-Bench: Advancing the Measurement of Text-Driven Image Editing for Human Perception Alignment [6.627422081288281]
  We introduce the Text-driven Image Editing Benchmark suite (IE-Bench) to enhance the assessment of text-driven edited images.<n>IE-Bench includes a database containing diverse source images, various editing prompts and the corresponding results.<n>We also introduce IE-QA, a multi-modality source-aware quality assessment method for text-driven image editing.
  arXiv  Detail & Related papers  (2025-01-17T02:47:25Z)
• Visual question answering based evaluation metrics for text-to-image generation [7.105786967332924]
  This paper proposes new evaluation metrics that assess the alignment between input text and generated images for every individual object. Experimental results show that our proposed evaluation approach is the superior metric that can simultaneously assess finer text-image alignment and image quality.
  arXiv  Detail & Related papers  (2024-11-15T13:32:23Z)
• ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models [11.830273909934688]
  Modern Text-to-Image (T2I) Diffusion models have revolutionized image editing by enabling the generation of high-quality images. We propose ReEdit, a modular and efficient end-to-end framework that captures edits in both text and image modalities. Our results demonstrate that ReEdit consistently outperforms contemporary approaches both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2024-11-06T15:19:24Z)
• Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editing [26.086806549826058]
  Text-guided image editing seeks the preservation of core elements in the source image while implementing modifications based on the target text.<n>Existing metrics have a context-blindness problem, indiscriminately applying the same evaluation criteria on completely different pairs of source image and target text.<n>We propose AugCLIP, a context-aware metric that adaptively coordinates preservation and modification aspects.
  arXiv  Detail & Related papers  (2024-10-15T08:12:54Z)
• Ground-A-Score: Scaling Up the Score Distillation for Multi-Attribute Editing [49.419619882284906]
  Ground-A-Score is a powerful model-agnostic image editing method by incorporating grounding during score distillation. The selective application with a new penalty coefficient and contrastive loss helps to precisely target editing areas. Both qualitative assessments and quantitative analyses confirm that Ground-A-Score successfully adheres to the intricate details of extended and multifaceted prompts.
  arXiv  Detail & Related papers  (2024-03-20T12:40:32Z)
• iEdit: Localised Text-guided Image Editing with Weak Supervision [53.082196061014734]
  We propose a novel learning method for text-guided image editing. It generates images conditioned on a source image and a textual edit prompt. It shows favourable results against its counterparts in terms of image fidelity, CLIP alignment score and qualitatively for editing both generated and real images.
  arXiv  Detail & Related papers  (2023-05-10T07:39:14Z)
• Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting [53.708523312636096]
  We present Imagen Editor, a cascaded diffusion model built, by fine-tuning on text-guided image inpainting. edits are faithful to the text prompts, which is accomplished by using object detectors to propose inpainting masks during training. To improve qualitative and quantitative evaluation, we introduce EditBench, a systematic benchmark for text-guided image inpainting.
  arXiv  Detail & Related papers  (2022-12-13T21:25:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.