UniREditBench: A Unified Reasoning-based Image Editing Benchmark

• URL: http://arxiv.org/abs/2511.01295v1
• Date: Mon, 03 Nov 2025 07:24:57 GMT
• Title: UniREditBench: A Unified Reasoning-based Image Editing Benchmark
• Abstract summary: This work proposes UniREditBench, a unified benchmark for reasoning-based image editing evaluation.<n>It comprises 2,700 meticulously curated samples, covering both real- and game-world scenarios across 8 primary dimensions and 18 sub-dimensions.<n>We fine-tune Bagel on this dataset and develop UniREdit-Bagel, demonstrating substantial improvements in both in-domain and out-of-distribution settings.
• Score: 52.54256348710893
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in multi-modal generative models have driven substantial improvements in image editing. However, current generative models still struggle with handling diverse and complex image editing tasks that require implicit reasoning, underscoring the need for a comprehensive benchmark to systematically assess their performance across various reasoning scenarios. Existing benchmarks primarily focus on single-object attribute transformation in realistic scenarios, which, while effective, encounter two key challenges: (1) they largely overlook multi-object interactions as well as game-world scenarios that involve human-defined rules, which are common in real-life applications; (2) they only rely on textual references to evaluate the generated images, potentially leading to systematic misjudgments, especially in complex reasoning scenarios. To this end, this work proposes UniREditBench, a unified benchmark for reasoning-based image editing evaluation. It comprises 2,700 meticulously curated samples, covering both real- and game-world scenarios across 8 primary dimensions and 18 sub-dimensions. To improve evaluation reliability, we introduce multimodal dual-reference evaluation, providing both textual and ground-truth image references for each sample assessment. Furthermore, we design an automated multi-scenario data synthesis pipeline and construct UniREdit-Data-100K, a large-scale synthetic dataset with high-quality chain-of-thought (CoT) reasoning annotations. We fine-tune Bagel on this dataset and develop UniREdit-Bagel, demonstrating substantial improvements in both in-domain and out-of-distribution settings. Through thorough benchmarking of both open-source and closed-source image editing models, we reveal their strengths and weaknesses across various aspects.

Related papers

• An Extensive Benchmark for Single-round and Multi-round Instruction-based Image Editing [121.27321922764145]
  We present I2EBench2.0, a comprehensive evaluation benchmark for instruction-based image editing (IIE) models.<n>I2EBench2.0 simultaneously evaluates both single-round and multi-round instruction-based edits.<n>It incorporates 16 dimensions for single-round evaluations and 7 for multi-round evaluations.
  arXiv  Detail & Related papers  (2026-06-14T03:26:56Z)
• Rethinking Scribble-Guided Image Editing: Generalization, Instruction Adherence, and Multi-Tasking [14.173267480504842]
  Scribble-guided image editing allows users to combine simple scribble annotations with text prompts to specify both where and how an image should be edited.<n>Existing models still exhibit unstable performance under this paradigm, especially in multi-task scenarios.<n>We conduct empirical studies using an open-source editing model and reveal an asymmetry in generalization.
  arXiv  Detail & Related papers  (2026-05-25T08:20:23Z)
• Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling [18.3502725927898]
  We introduce Edit- and Edit-Reward, a unified evaluation suite for image editing and reward modeling.<n> Edit-Reward includes 2,388 annotated instances spanning six progressively challenging task categories.<n>It adopts a fine-grained multidimensional evaluation framework based on structured reasoning and carefully designed rubrics.
  arXiv  Detail & Related papers  (2026-05-13T06:33:54Z)
• A Sanity Check on Composed Image Retrieval [91.95275287747499]
  Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification.<n>We introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs.<n>We propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios.
  arXiv  Detail & Related papers  (2026-04-14T15:52:22Z)
• DLEBench: Evaluating Small-scale Object Editing Ability for Instruction-based Image Editing Model [10.609050605838805]
  This paper introduces DeepLookEditBench, the first benchmark dedicated to assessing the abilities of IIEMs in editing small-scale objects.<n>We construct a challenging testbed comprising 1889 samples across seven instruction types.<n>In these samples, target objects occupy only 1%-10% of the image area, covering complex scenarios such as partial occlusion and multi-object editing.<n> Empirical results on 10 IIEMs reveal significant performance gaps in small-scale object editing, highlighting the need for specialized benchmarks to advance this ability.
  arXiv  Detail & Related papers  (2026-02-27T02:59:34Z)
• I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models [78.62380562116135]
  Existing image editing benchmarks suffer from limited task scopes, insufficient evaluation dimensions, and heavy reliance on manual annotations.<n>We propose textbfI2I-Bench, a comprehensive benchmark for image-to-image editing models, which features 10 task categories across both single-image and multi-image editing tasks.<n>Using I2I-Bench, we benchmark numerous mainstream image editing models, investigating the gaps and trade-offs between editing models across various dimensions.
  arXiv  Detail & Related papers  (2025-12-04T10:44:07Z)
• The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignment [105.31858867473845]
  ImageCritic can be integrated into an agent framework to automatically detect inconsistencies and correct them with multi-round and local editing.<n>In experiments, ImageCritic can effectively resolve detail-related issues in various customized generation scenarios, providing significant improvements over existing methods.
  arXiv  Detail & Related papers  (2025-11-25T18:40:25Z)
• Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content [71.46991494014382]
  We introduce Q-Real, a novel dataset for fine-grained evaluation of realism and plausibility in AI-generated images.<n>Q-Real consists of 3,088 images generated by popular text-to-image models.<n>We construct Q-Real Bench to evaluate them on two tasks: judgment and grounding with reasoning.
  arXiv  Detail & Related papers  (2025-11-21T02:43:17Z)
• MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval [86.35779264575154]
  Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios.<n>We introduce MR$2$-Bench, a reasoning-intensive benchmark for multimodal retrieval.
  arXiv  Detail & Related papers  (2025-09-30T15:09:14Z)
• Why Settle for One? Text-to-ImageSet Generation and Evaluation [72.55708276046124]
  Text-to-ImageSet (T2IS) generation aims to generate sets of images that meet various consistency requirements based on user instructions.<n>We propose $textbfAutoT2IS$, a training-free framework that maximally leverages pretrained Transformers' in-context capabilities to harmonize visual elements.<n>Our method also demonstrates the ability to enable numerous underexplored real-world applications, confirming its substantial practical value.
  arXiv  Detail & Related papers  (2025-06-29T15:01:16Z)
• OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation [23.05106664412349]
  Text-to-image (T2I) models have garnered significant attention for generating high-quality images aligned with text prompts.<n>OneIG-Bench is a benchmark framework for evaluation of T2I models across multiple dimensions.
  arXiv  Detail & Related papers  (2025-06-09T17:50:21Z)
• What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Models [88.398085358514]
  DICE is a model designed to detect localized differences between the original and the edited image.<n>It is trained using a strategy that leverages self-supervision, distillation from inpainting networks, and full supervision.<n>We demonstrate that DICE effectively identifies coherent edits, effectively evaluating images generated by different editing models with a strong correlation with human judgment.
  arXiv  Detail & Related papers  (2025-05-26T18:00:10Z)
• ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing [23.512687688393346]
  ICE-Bench is a comprehensive benchmark designed to rigorously assess image generation models.<n>The evaluation framework assesses image generation capabilities across 6 dimensions.<n>We conduct a thorough analysis of existing generation models, revealing both the challenging nature of our benchmark and the gap between current model capabilities and real-world generation requirements.
  arXiv  Detail & Related papers  (2025-03-18T17:53:29Z)
• PixLens: A Novel Framework for Disentangled Evaluation in Diffusion-Based Image Editing with Object Detection + SAM [17.89238060470998]
  evaluating diffusion-based image-editing models is a crucial task in the field of Generative AI. Our benchmark, PixLens, provides a comprehensive evaluation of both edit quality and latent representation disentanglement.
  arXiv  Detail & Related papers  (2024-10-08T06:05:15Z)
• I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing [67.05794909694649]
  We propose I2EBench, a comprehensive benchmark to evaluate the quality of edited images produced by IIE models. I2EBench consists of 2,000+ images for editing, along with 4,000+ corresponding original and diverse instructions. We will open-source I2EBench, including all instructions, input images, human annotations, edited images from all evaluated methods, and a simple script for evaluating the results from new IIE models.
  arXiv  Detail & Related papers  (2024-08-26T11:08:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.