Omni IIE Bench: Benchmarking the Practical Capabilities of Image Editing Models

• URL: http://arxiv.org/abs/2603.16944v1
• Date: Mon, 16 Mar 2026 08:07:06 GMT
• Title: Omni IIE Bench: Benchmarking the Practical Capabilities of Image Editing Models
• Abstract summary: We introduce Omni IIE Bench, a benchmark designed to diagnose the editing consistency of IIE models in practical application scenarios.<n>We perform a comprehensive evaluation of 8 mainstream IIE models using Omni IIE Bench.<n>Our analysis quantifies, for the first time, a prevalent performance gap when transitioning from low-semantic-scale to high-semantic-scale tasks.
• Score: 12.603176617170504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Instruction-based Image Editing (IIE) has achieved significant progress, existing benchmarks pursue task breadth via mixed evaluations. This paradigm obscures a critical failure mode crucial in professional applications: the inconsistent performance of models across tasks of varying semantic scales. To address this gap, we introduce Omni IIE Bench, a high-quality, human-annotated benchmark specifically designed to diagnose the editing consistency of IIE models in practical application scenarios. Omni IIE Bench features an innovative dual-track diagnostic design: (1) Single-turn Consistency, comprising shared-context task pairs of attribute modification and entity replacement; and (2) Multi-turn Coordination, involving continuous dialogue tasks that traverse semantic scales. The benchmark is constructed via an exceptionally rigorous multi-stage human filtering process, incorporating a quality standard enforced by computer vision graduate students and an industry relevance review conducted by professional designers. We perform a comprehensive evaluation of 8 mainstream IIE models using Omni IIE Bench. Our analysis quantifies, for the first time, a prevalent performance gap: nearly all models exhibit a significant performance degradation when transitioning from low-semantic-scale to high-semantic-scale tasks. Omni IIE Bench provides critical diagnostic tools and insights for the development of next-generation, more reliable, and stable IIE models.

Related papers

• An Extensive Benchmark for Single-round and Multi-round Instruction-based Image Editing [121.27321922764145]
  We present I2EBench2.0, a comprehensive evaluation benchmark for instruction-based image editing (IIE) models.<n>I2EBench2.0 simultaneously evaluates both single-round and multi-round instruction-based edits.<n>It incorporates 16 dimensions for single-round evaluations and 7 for multi-round evaluations.
  arXiv  Detail & Related papers  (2026-06-14T03:26:56Z)
• DSAINet: An Efficient Dual-Scale Attentive Interaction Network for General EEG Decoding [10.266590707498029]
  DSAINet is an efficient dual-scale interaction network for general EEG decoding.<n>It consistently outperforms 13 representative baselines under strict subject-independent evaluation.<n>It achieves a favorable accuracy-efficiency trade-off with only about 77K trainable parameters.
  arXiv  Detail & Related papers  (2026-04-20T11:10:33Z)
• AIDABench: AI Data Analytics Benchmark [62.45908988324612]
  AIDABench is a benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner.<n>AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation.<n>We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families.
  arXiv  Detail & Related papers  (2026-02-27T08:58:05Z)
• AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems [52.65695508605237]
  We introduce AI-NativeBench, the first application-centric and white-box AI-Native benchmark suite grounded in Model Context Protocol (MCP) and Agent-to-Agent (A2A) standards.<n>By treating agentic spans as first-class citizens within distributed traces, our methodology enables granular analysis of engineering characteristics beyond simple capabilities.<n>This work provides the first systematic evidence to guide the transition from measuring model capability to engineering reliable AI-Native systems.
  arXiv  Detail & Related papers  (2026-01-14T11:32:07Z)
• ICAD-LLM: One-for-All Anomaly Detection via In-Context Learning with Large Language Models [14.804039283733475]
  Anomaly detection is a fundamental task of critical importance across numerous domains.<n>We introduce a novel paradigm: In-Context Anomaly Detection (ICAD), where anomalies are defined by their dissimilarity to a relevant reference set of normal samples.<n>Under this paradigm, we propose ICAD-LLM, a unified AD framework leveraging Large Language Models' in-context learning abilities to process heterogeneous data within a single model.
  arXiv  Detail & Related papers  (2025-12-01T13:41:30Z)
• WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality [62.43165871914528]
  We introduce WebDevJudge, a systematic benchmark for assessing LLM-as-a-judge performance in web development.<n>WebDevJudge comprises human preference labels over paired web implementations, annotated with structured and query-grounded rubrics.<n>In-depth analysis indicates this gap stems from fundamental model limitations, including failures in recognizing functional equivalence, verifying task feasibility, and mitigating bias.
  arXiv  Detail & Related papers  (2025-10-21T12:16:04Z)
• WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning [24.178675410636135]
  We present a large-scale benchmark of 45.1k webpages collected from real-world portal sites.<n>We also propose a novel evaluation metric that measures layout and style consistency from the final rendered pages.
  arXiv  Detail & Related papers  (2025-10-05T08:47:39Z)
• ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation [51.297873393639456]
  ArtifactsBench is a framework for automated visual code generation evaluation.<n>Our framework renders each generated artifact and captures its dynamic behavior through temporal screenshots.<n>We construct a new benchmark of 1,825 diverse tasks and evaluate over 30 leading Large Language Models.
  arXiv  Detail & Related papers  (2025-07-07T12:53:00Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)
• I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing [67.05794909694649]
  We propose I2EBench, a comprehensive benchmark to evaluate the quality of edited images produced by IIE models. I2EBench consists of 2,000+ images for editing, along with 4,000+ corresponding original and diverse instructions. We will open-source I2EBench, including all instructions, input images, human annotations, edited images from all evaluated methods, and a simple script for evaluating the results from new IIE models.
  arXiv  Detail & Related papers  (2024-08-26T11:08:44Z)
• Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment [38.574204922793626]
  We present a further analysis of visual modality incompleteness, benchmarking latest MMEA models on our proposed dataset MMEA-UMVM. Our research indicates that, in the face of modality incompleteness, models succumb to overfitting the modality noise, and exhibit performance oscillations or declines at high rates of missing modality. We introduce UMAEA, a robust multi-modal entity alignment approach designed to tackle uncertainly missing and ambiguous visual modalities.
  arXiv  Detail & Related papers  (2023-07-30T12:16:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.