StableI2I: Spotting Unintended Changes in Image-to-Image Transition

• URL: http://arxiv.org/abs/2605.04453v1
• Date: Wed, 06 May 2026 03:28:05 GMT
• Title: StableI2I: Spotting Unintended Changes in Image-to-Image Transition
• Abstract summary: We propose StableI2I, a unified and dynamic evaluation framework that measures content fidelity and pre-post consistency.<n>Our framework serves as a practical and reliable evaluation tool for diagnosing content consistency and benchmarking model performance in real-world I2I systems.
• Score: 35.72064037549274
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In most real-world image-to-image (I2I) scenarios, existing evaluations primarily focus on instruction following and the perceptual quality or aesthetics of the generated images. However, they largely fail to assess whether the output image preserves the semantic correspondence and spatial structure of the input image. To address this limitation, we propose StableI2I, a unified and dynamic evaluation framework that explicitly measures content fidelity and pre--post consistency across a wide range of I2I tasks without requiring reference images, including image editing and image restoration. In addition, we construct StableI2I-Bench, a benchmark designed to systematically evaluate the accuracy of MLLMs on such fidelity and consistency assessment tasks. Extensive experimental results demonstrate that StableI2I provides accurate, fine-grained, and interpretable evaluations of content fidelity and consistency, with strong correlations to human subjective judgments. Our framework serves as a practical and reliable evaluation tool for diagnosing content consistency and benchmarking model performance in real-world I2I systems.

Related papers

• Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation [50.85337196571056]
  Text-to-image (T2I) generation has advanced rapidly, making reliable evaluation critical as performance differences between models narrow.<n>Existing evaluation practices typically apply uniform annotation mechanisms, such as Likert-scale or binary question answering (BQA)<n>We revisit T2I evaluation through the lens of skill-aligned annotation, where annotation strategies reflect the underlying characteristics of each evaluation skill.
  arXiv  Detail & Related papers  (2026-05-13T09:14:31Z)
• DiffCap-Bench: A Comprehensive, Challenging, Robust Benchmark for Image Difference Captioning [26.88130913151649]
  Image Difference Captioning (IDC) generates language descriptions that precisely identify differences between two images.<n>We introduce DiffCap-Bench, a comprehensive IDC benchmark covering ten distinct difference categories.<n>We also propose an LLM-as-a-Judge evaluation protocol grounded in human-validated Difference Lists.
  arXiv  Detail & Related papers  (2026-05-06T05:12:41Z)
• A Sanity Check on Composed Image Retrieval [91.95275287747499]
  Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the desired modification.<n>We introduce FISD, a Fully-Informed Semantically-Diverse benchmark, which employs generative models to precisely control the variables of reference-target image pairs.<n>We propose an automatic multi-round agentic evaluation framework to probe the potential of the existing models in the interactive scenarios.
  arXiv  Detail & Related papers  (2026-04-14T15:52:22Z)
• Redefining Quality Criteria and Distance-Aware Score Modeling for Image Editing Assessment [28.60763581601123]
  IEQA requires complex reasoning over multimodal inputs and multi-dimensional assessments.<n>We propose Define-and-Score Image Editing Quality Assessment (DS-IEQA), a unified framework that jointly learns evaluation criteria and score representations.<n>Our method ranks 4th in the 2026 NTIRE X-AIGC Quality Assessment Track 2 without any additional training data.
  arXiv  Detail & Related papers  (2026-04-14T01:05:41Z)
• SIQA: Toward Reliable Scientific Image Quality Assessment [72.41803245808924]
  We introduce Scientific Image Quality Assessment (SIQA), a framework that models scientific image quality along two complementary dimensions.<n>We design two evaluation protocols: SIQA-U (Understanding), which measures semantic comprehension of scientific content through multiple-choice tasks, and SIQA-S (Scoring), which evaluates alignment with expert quality judgments.<n>Experiments across representative multimodal large language models (MLLMs) reveal a consistent discrepancy between scoring alignment and scientific understanding.
  arXiv  Detail & Related papers  (2026-03-05T06:57:26Z)
• I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models [78.62380562116135]
  Existing image editing benchmarks suffer from limited task scopes, insufficient evaluation dimensions, and heavy reliance on manual annotations.<n>We propose textbfI2I-Bench, a comprehensive benchmark for image-to-image editing models, which features 10 task categories across both single-image and multi-image editing tasks.<n>Using I2I-Bench, we benchmark numerous mainstream image editing models, investigating the gaps and trade-offs between editing models across various dimensions.
  arXiv  Detail & Related papers  (2025-12-04T10:44:07Z)
• Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content [71.46991494014382]
  We introduce Q-Real, a novel dataset for fine-grained evaluation of realism and plausibility in AI-generated images.<n>Q-Real consists of 3,088 images generated by popular text-to-image models.<n>We construct Q-Real Bench to evaluate them on two tasks: judgment and grounding with reasoning.
  arXiv  Detail & Related papers  (2025-11-21T02:43:17Z)
• UniREditBench: A Unified Reasoning-based Image Editing Benchmark [52.54256348710893]
  This work proposes UniREditBench, a unified benchmark for reasoning-based image editing evaluation.<n>It comprises 2,700 meticulously curated samples, covering both real- and game-world scenarios across 8 primary dimensions and 18 sub-dimensions.<n>We fine-tune Bagel on this dataset and develop UniREdit-Bagel, demonstrating substantial improvements in both in-domain and out-of-distribution settings.
  arXiv  Detail & Related papers  (2025-11-03T07:24:57Z)
• OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation [23.05106664412349]
  Text-to-image (T2I) models have garnered significant attention for generating high-quality images aligned with text prompts.<n>OneIG-Bench is a benchmark framework for evaluation of T2I models across multiple dimensions.
  arXiv  Detail & Related papers  (2025-06-09T17:50:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.