Evaluating Image Editing with LLMs: A Comprehensive Benchmark and Intermediate-Layer Probing Approach

• URL: http://arxiv.org/abs/2603.19775v2
• Date: Wed, 25 Mar 2026 02:23:16 GMT
• Title: Evaluating Image Editing with LLMs: A Comprehensive Benchmark and Intermediate-Layer Probing Approach
• Abstract summary: We introduce TIEdit, a benchmark for systematic evaluation of text-guided image editing methods.<n>To obtain reliable subjective ratings, 20 experts are recruited to produce 307,200 raw subjective ratings.<n>We further propose EditProbe, an LLM-based evaluator that estimates editing quality via intermediate-layer probing of hidden representations.
• Score: 48.01137214342501
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating text-guided image editing (TIE) methods remains a challenging problem, as reliable assessment should simultaneously consider perceptual quality, alignment with textual instructions, and preservation of original image content. Despite rapid progress in TIE models, existing evaluation benchmarks remain limited in scale and often show weak correlation with human perceptual judgments. In this work, we introduce TIEdit, a benchmark for systematic evaluation of text-guided image editing methods. TIEdit consists of 512 source images paired with editing prompts across eight representative editing tasks, producing 5,120 edited images generated by ten state-of-the-art TIE models. To obtain reliable subjective ratings, 20 experts are recruited to produce 307,200 raw subjective ratings, which accumulates into 15,360 mean opinion scores (MOSs) across three evaluation dimensions: perceptual quality, editing alignment, and content preservation. Beyond the benchmark itself, we further propose EditProbe, an LLM-based evaluator that estimates editing quality via intermediate-layer probing of hidden representations. Instead of relying solely on final model outputs, EditProbe extracts informative representations from intermediate layers of multimodal large language models to better capture semantic and perceptual relationships between source images, editing instructions, and edited results. Experimental results demonstrate that widely used automatic evaluation metrics show limited correlation with human judgments on editing tasks, while EditProbe achieves substantially stronger alignment with human perception. Together, TIEdit and EditProbe provide a foundation for more reliable and perceptually aligned evaluation of text-guided image editing methods.

Related papers

• ReasonEdit: Towards Interpretable Image Editing Evaluation via Reinforcement Learning [86.61218827780675]
  We present ReasonEdit, an evaluation tool for text-guided image editing.<n>It is trained using reward signals derived from RE-Reward and the Group Relative Policy Optimization (GRPO) algorithm.<n>It is capable of generating high-quality interpretable evaluation text, enabling more transparent and trustworthy assessment for image editing.
  arXiv  Detail & Related papers  (2026-05-08T09:23:26Z)
• EditRefiner: A Human-Aligned Agentic Framework for Image Editing Refinement [76.76247443244293]
  EditRefiner is an agentic framework that reformulates post-editing correction as a human-like perception-reasoning-action-evaluation loop.<n>It consistently outperforms state-of-the-art methods in distortion, diagnose accuracy and human perception alignment.
  arXiv  Detail & Related papers  (2026-05-08T09:05:08Z)
• Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis [95.89328387635176]
  We introduce a fine-grained Multimodal Large Language Model (MLLM)-as-a-Judge framework for image editing.<n>We present a new human-validated benchmark that integrates human judgments, MLLM-based evaluations, model outputs, and traditional metrics.
  arXiv  Detail & Related papers  (2026-02-13T15:34:32Z)
• IE-Critic-R1: Advancing the Explanatory Measurement of Text-Driven Image Editing for Human Perception Alignment [14.001770505266116]
  We introduce the Text-driven Image Editing Benchmark suite (IE-Bench) to enhance the assessment of text-driven edited images.<n>IE-Bench includes a database of diverse source images, various editing prompts and the corresponding edited results from different editing methods.<n>IE-Critic-R1 provides more comprehensive and explainable quality assessment for text-driven image editing that aligns with human perception.
  arXiv  Detail & Related papers  (2025-11-22T13:16:58Z)
• LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs [104.97913324464247]
  We introduce EBench-18K, the first large-scale image Editing Benchmark including 18K edited images with fine-grained human preference annotations.<n>EBench-18K includes 1,080 source images with corresponding editing prompts across 21 tasks, 18K+ edited images produced by 17 state-of-the-art TIE models, 55K+ mean opinion scores (MOSs) assessed from three evaluation dimensions, and 18K+ question-answering (QA) pairs.<n>Then, we propose LMM4Edit, a LMM-based metric for evaluating image Editing models from perceptual quality, editing alignment, attribute preservation
  arXiv  Detail & Related papers  (2025-07-22T03:11:07Z)
• GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing [60.66800567924348]
  We introduce a new benchmark designed to evaluate text-guided image editing models.<n>The benchmark includes over 1000 high-quality editing examples across 20 diverse content categories.<n>We conduct a large-scale study comparing GPT-Image-1 against several state-of-the-art editing models.
  arXiv  Detail & Related papers  (2025-05-16T17:55:54Z)
• PixLens: A Novel Framework for Disentangled Evaluation in Diffusion-Based Image Editing with Object Detection + SAM [17.89238060470998]
  evaluating diffusion-based image-editing models is a crucial task in the field of Generative AI. Our benchmark, PixLens, provides a comprehensive evaluation of both edit quality and latent representation disentanglement.
  arXiv  Detail & Related papers  (2024-10-08T06:05:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.