Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling

• URL: http://arxiv.org/abs/2605.13062v1
• Date: Wed, 13 May 2026 06:33:54 GMT
• Title: Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling
• Abstract summary: We introduce Edit- and Edit-Reward, a unified evaluation suite for image editing and reward modeling.<n> Edit-Reward includes 2,388 annotated instances spanning six progressively challenging task categories.<n>It adopts a fine-grained multidimensional evaluation framework based on structured reasoning and carefully designed rubrics.
• Score: 18.3502725927898
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent image editing models have achieved remarkable progress in instruction following, multimodal understanding, and complex visual editing. However, existing benchmarks often fail to faithfully reflect human judgment, especially for strong frontier models, due to limited task difficulty and coarse-grained evaluation protocols. In parallel, reward models have become increasingly important for RL-based image editing optimization, yet existing reward model benchmarks still rely on unrealistic evaluation settings that deviate from practical RL scenarios. These limitations hinder reliable assessment of both image editing models and reward models. To address these challenges, we introduce Edit-Compass and EditReward-Compass, a unified evaluation suite for image editing and reward modeling. Edit-Compass contains 2,388 carefully annotated instances spanning six progressively challenging task categories, covering capabilities such as world knowledge reasoning, visual reasoning, and multi-image editing. Beyond broad task coverage, Edit-Compass adopts a fine-grained multidimensional evaluation framework based on structured reasoning and carefully designed scoring rubrics. In parallel, EditReward-Compass contains 2,251 preference pairs that simulate realistic reward modeling scenarios during RL optimization.

Related papers

• ReasonEdit: Towards Interpretable Image Editing Evaluation via Reinforcement Learning [86.61218827780675]
  We present ReasonEdit, an evaluation tool for text-guided image editing.<n>It is trained using reward signals derived from RE-Reward and the Group Relative Policy Optimization (GRPO) algorithm.<n>It is capable of generating high-quality interpretable evaluation text, enabling more transparent and trustworthy assessment for image editing.
  arXiv  Detail & Related papers  (2026-05-08T09:23:26Z)
• Making Image Editing Easier via Adaptive Task Reformulation with Agentic Executions [27.891822809255682]
  We propose an adaptive task reformulation framework that improves editing performance without modifying the underlying model.<n>Our key idea is to transform the original image-instruction pair into a sequence of operations that are dynamically determined and executed by a MLLM agent.
  arXiv  Detail & Related papers  (2026-04-17T10:17:22Z)
• GEditBench v2: A Human-Aligned Benchmark for General Image Editing [58.86807672117726]
  We introduce GEditBench v2, a comprehensive benchmark with 1,200 real-world user queries spanning 23 tasks.<n>We also propose PVC-Judge, an open-source pairwise assessment model for visual consistency, trained via two novel region-decoupled preference data synthesis pipelines.<n> PVC-Judge achieves state-of-the-art evaluation performance among open-source models and even surpasses GPT-5.1 on average.
  arXiv  Detail & Related papers  (2026-03-30T15:08:32Z)
• Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation [67.26349227500084]
  Current reward models, which act as critics during reinforcement learning, often suffer from hallucinations and assign noisy scores.<n>We present FIRM, a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing.<n> FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models.
  arXiv  Detail & Related papers  (2026-03-12T17:57:21Z)
• DLEBench: Evaluating Small-scale Object Editing Ability for Instruction-based Image Editing Model [10.609050605838805]
  This paper introduces DeepLookEditBench, the first benchmark dedicated to assessing the abilities of IIEMs in editing small-scale objects.<n>We construct a challenging testbed comprising 1889 samples across seven instruction types.<n>In these samples, target objects occupy only 1%-10% of the image area, covering complex scenarios such as partial occlusion and multi-object editing.<n> Empirical results on 10 IIEMs reveal significant performance gaps in small-scale object editing, highlighting the need for specialized benchmarks to advance this ability.
  arXiv  Detail & Related papers  (2026-02-27T02:59:34Z)
• I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models [78.62380562116135]
  Existing image editing benchmarks suffer from limited task scopes, insufficient evaluation dimensions, and heavy reliance on manual annotations.<n>We propose textbfI2I-Bench, a comprehensive benchmark for image-to-image editing models, which features 10 task categories across both single-image and multi-image editing tasks.<n>Using I2I-Bench, we benchmark numerous mainstream image editing models, investigating the gaps and trade-offs between editing models across various dimensions.
  arXiv  Detail & Related papers  (2025-12-04T10:44:07Z)
• UniREditBench: A Unified Reasoning-based Image Editing Benchmark [52.54256348710893]
  This work proposes UniREditBench, a unified benchmark for reasoning-based image editing evaluation.<n>It comprises 2,700 meticulously curated samples, covering both real- and game-world scenarios across 8 primary dimensions and 18 sub-dimensions.<n>We fine-tune Bagel on this dataset and develop UniREdit-Bagel, demonstrating substantial improvements in both in-domain and out-of-distribution settings.
  arXiv  Detail & Related papers  (2025-11-03T07:24:57Z)
• EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling [71.8265422228785]
  Reinforcement Learning (RL) offers a promising solution, but its adoption in image editing has been hindered by the lack of a high-fidelity, efficient reward signal.<n>We present a comprehensive methodology to overcome this barrier, centered on the development of a state-of-the-art, specialized reward model.
  arXiv  Detail & Related papers  (2025-09-28T14:28:24Z)
• PixLens: A Novel Framework for Disentangled Evaluation in Diffusion-Based Image Editing with Object Detection + SAM [17.89238060470998]
  evaluating diffusion-based image-editing models is a crucial task in the field of Generative AI. Our benchmark, PixLens, provides a comprehensive evaluation of both edit quality and latent representation disentanglement.
  arXiv  Detail & Related papers  (2024-10-08T06:05:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.