MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos

• URL: http://arxiv.org/abs/2607.00491v1
• Date: Wed, 01 Jul 2026 06:19:54 GMT
• Title: MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos
• Abstract summary: MindEdit-Bench is a benchmark of six spatial reasoning tasks built from three-photo smartphone triplets of newly captured indoor scenes.<n>Each question provides 8-24 structured answer choices, enabling answer-letter-level diagnosis of spatial and fallback errors.<n>The benchmark covers 120 private indoor scenes not drawn from public datasets.
• Score: 20.94052676416818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Benchmarks for vision-language models (VLMs) mostly test observational spatial reasoning: models describe relations already visible in the input. Existing what-if tasks typically vary the observer while keeping the scene fixed. Can VLMs instead predict the consequences of hypothetically moving or rotating an object? We introduce MindEdit-Bench, a benchmark of six spatial reasoning tasks built from three-photo smartphone triplets of newly captured indoor scenes via an automatic in-the-wild 3D scene-graph extraction pipeline. Four tasks probe perception and perspective transformation over observed structure; two new tasks, L4 (spatial editing) and L5 (cross-view visibility editing), probe object-level counterfactual reasoning, where correct answers are absent from all input images. Each question provides 8-24 structured answer choices, enabling answer-letter-level diagnosis of spatial and fallback errors. The benchmark covers 120 private indoor scenes not drawn from public datasets, reducing public-data pretraining-overlap risk. Across 15 VLMs on 1,003 human-verified questions, task-wise mean VLM accuracy is only 8%-31%, versus 81%-97% human majority-vote accuracy. The pooled human--best-VLM gap is 53 pp, with at least 39 pp on every task. The structured answer space further reveals non-uniform failures, including weaker camera-depth-axis inference and fallback behavior on difficult visibility-editing cases.

Related papers

• When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning [73.56343820592399]
  We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed.<n>Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average.<n>We present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view.
  arXiv  Detail & Related papers  (2026-08-05T11:46:15Z)
• OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs [49.82435858489898]
  OVOS-Bench is a fully-annotated benchmark for streaming human-centric spatial intelligence.<n>It consists of 1,680 questions on 348 source videos.<n>Each question carries a query and an evidence interval, and at evaluation, the model sees only the prefix preceding the query.
  arXiv  Detail & Related papers  (2026-06-02T16:51:32Z)
• ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search [49.61286310968402]
  We introduce ARGOS, the first benchmark and framework that reformulates multi-camera person search as an interactive reasoning problem.<n>An ARGOS agent receives a vague witness statement and must decide what to ask, when to invoke spatial or temporal tools, and how to interpret ambiguous responses.<n>The benchmark comprises 2,691 tasks across 14 real-world scenarios in three progressive tracks: semantic perception (Who), spatial reasoning (Where), and temporal reasoning (When)
  arXiv  Detail & Related papers  (2026-04-14T14:06:19Z)
• SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild? [11.64929380114363]
  SpatiaLab comprises 1,400 visual question-answer pairs across six major categories.<n>Each subcategory contains at least 25 questions, and each main category includes at least 200 questions.<n>In the multiple-choice setup, InternVL3.5-72B achieves 54.93% accuracy versus 87.57% for humans.
  arXiv  Detail & Related papers  (2026-02-03T17:52:02Z)
• Tri-Bench: Stress-Testing VLM Reliability on Spatial Reasoning under Camera Tilt and Object Interference [0.0]
  Verifiable geometric reasoning is a critical component for trustworthy and controllable agentic AI.<n>We present Tri-Bench, a compact benchmark of planar triangle problems.<n>We evaluate four recent Vision-Language Models (VLMs) using a single, fixed prompt.
  arXiv  Detail & Related papers  (2025-12-09T17:52:57Z)
• VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes [36.370533774426555]
  We present VisualOverload, a visual question answering (VQA) benchmark comprising 2,720 question-answer pairs.<n>Unlike prior VQA datasets that typically focus on near global image understanding, VisualOverload challenges models to perform simple, knowledge-free vision tasks in densely populated scenes.<n>We observe that even the best model (o3) out of 37 tested models only achieves 19.6% accuracy on our hardest test split and overall 69.5% accuracy on all questions.
  arXiv  Detail & Related papers  (2025-09-29T18:00:25Z)
• Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology [87.65242416688146]
  TreeBench is a diagnostic benchmark for visual grounded reasoning.<n>TreeVGR is a training paradigm to supervise localization and reasoning jointly with reinforcement learning.
  arXiv  Detail & Related papers  (2025-07-10T17:59:58Z)
• MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence [74.51213082084428]
  MMSI-Bench is a VQA benchmark dedicated to multi-image spatial intelligence.<n>We conduct extensive experiments and thoroughly evaluate 34 open-source and proprietary MLLMs.<n>The strongest open-source model attains roughly 30% accuracy and OpenAI's o3 reasoning model reaches 40%, while humans score 97%.
  arXiv  Detail & Related papers  (2025-05-29T17:59:52Z)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models [59.0256377330646]
  Lens is a benchmark with 3.4K contemporary images and 60K+ human-authored questions covering eight tasks and 12 daily scenarios.<n>This dataset intrinsically supports to evaluate MLLMs to handle image-invariable prompts, from basic perception to compositional reasoning.<n>We evaluate 15+ frontier MLLMs such as Qwen2.5-VL-72B, InternVL3-78B, GPT-4o and two reasoning models QVQ-72B-preview and Kimi-VL.
  arXiv  Detail & Related papers  (2025-05-21T15:06:59Z)
• MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning [22.440669015518015]
  We evaluate whether multi-modal large language models (MLLMs) possess abstract visual reasoning abilities. Similar to the Sudoku puzzles, abstract visual reasoning (AVR) problems require finding high-level patterns. We introduce MARVEL, a benchmark with 770 MLLMs composed of six core knowledge patterns, geometric and abstract shapes, and five different task configurations.
  arXiv  Detail & Related papers  (2024-04-21T09:15:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.