VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

• URL: http://arxiv.org/abs/2609.06245v1
• Date: Sat, 05 Sep 2026 20:20:24 GMT
• Title: VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification
• Abstract summary: VDiff-Bench is a benchmark for fine-grained Image Difference Identification.<n>Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor.<n>Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle.
• Score: 53.25222629244709
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often struggle with a basic comparative skill: identifying what has changed between two similar images. We introduce VDiff-Bench, a challenging multiple-choice benchmark for fine-grained Image Difference Identification. VDiff-Bench contains 1,756 four-way questions over image pairs and covers 10 change categories: position, motion, regional image color, overall image color, appearance/disappearance, noise/resolution, texture, substitution/size, OCR/text, and illumination. Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor. To make the task challenging, we specifically curate ground-truth-conditioned negatives that require models to distinguish the actual change from nearby semantic alternatives. Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle: models exhibit uneven performance across sources and change categories, with persistent failures on subtle low-level changes like noises and textures. For instance, three 7-8B-scale open-source MLLMs score 52.5-70.6% on semantic changes but only 8.7-33.3% on low-level changes like noise and texture, falsely assuming no changes between two image inputs. Surprisingly, despite strong performance of other closed-source commercial models, Grok 4.3 demonstrate remarkable performance drop on identifying noise and texture differences between images, falling significantly behind large open-source models like Kimi K2.5 and K3. Overall, VDiff-Bench provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs, exposing failures that are not captured by standard single-image vision-language tasks.

Related papers

• DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues [54.31138496553705]
  DiCoBench is a comprehensive, multi-image high-resolution benchmark for fine-grained perception.<n>Our evaluation reveals a striking performance gap compared to human accuracy (98.3%)<n>We believe DiCoBench will serve as a challenging testbed to drive future research in autonomous, high-resolution multi-image perception.
  arXiv  Detail & Related papers  (2026-06-25T05:02:38Z)
• Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs [23.958966900531692]
  Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens.<n>We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes.
  arXiv  Detail & Related papers  (2026-03-10T02:14:23Z)
• Change3D: Revisiting Change Detection and Captioning from A Video Modeling Perspective [14.128228451821712]
  We present Change3D, a framework that reconceptualizes the change detection and captioning tasks through video modeling.<n>By integrating learnable perception frames between the bi-temporal images, a video encoder enables the perception frames to interact with the images directly and perceive their differences.
  arXiv  Detail & Related papers  (2025-03-24T15:48:07Z)
• Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation [13.311411816150551]
  We evaluate how well an MLLM understands a specific visual concept by its ability to uniquely describe two extremely similar images. We curate 247 highly similar image pairs as part of the D3 benchmark. For each image pair, the model is prompted to: (1) Detect a specific visual difference, and (2) Describe the target image uniquely such that it (3) Discriminates the target image from the distractor.
  arXiv  Detail & Related papers  (2024-09-23T15:31:25Z)
• Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning [71.14084801851381]
  Change captioning aims to succinctly describe the semantic change between a pair of similar images. Most existing methods directly capture the difference between them, which risk obtaining error-prone difference features. We propose a distractors-immune representation learning network that correlates the corresponding channels of two image representations.
  arXiv  Detail & Related papers  (2024-07-16T13:00:33Z)
• Describing Differences in Image Sets with Natural Language [101.80939666230168]
  Discerning set-level differences is crucial for understanding model behaviors and analyzing datasets. We introduce VisDiff, which first captions the images and prompts a language model to propose difference descriptions. We are able to find interesting and previously unknown differences in datasets and models, demonstrating VisDiff's utility in revealing nuanced insights.
  arXiv  Detail & Related papers  (2023-12-05T18:59:16Z)
• The Change You Want to See (Now in 3D) [65.61789642291636]
  The goal of this paper is to detect what has changed, if anything, between two "in the wild" images of the same 3D scene. We contribute a change detection model that is trained entirely on synthetic data and is class-agnostic. We release a new evaluation dataset consisting of real-world image pairs with human-annotated differences.
  arXiv  Detail & Related papers  (2023-08-21T01:59:45Z)
• Equivariant Similarity for Vision-Language Foundation Models [134.77524524140168]
  This study focuses on the multimodal similarity function that is not only the major training objective but also the core delivery to support downstream tasks. We propose EqSim, a regularization loss that can be efficiently calculated from any two matched training pairs. Compared to the existing evaluation sets, EqBen is the first to focus on "visual-minimal change"
  arXiv  Detail & Related papers  (2023-03-25T13:22:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.