ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning

• URL: http://arxiv.org/abs/2604.24300v2
• Date: Tue, 05 May 2026 23:43:25 GMT
• Title: ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning
• Abstract summary: Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
• Score: 59.558706734431276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings. First, many benchmarks derive question-answer (QA) pairs from point-cloud-based 3D annotations originally curated for traditional 3D perception. When such annotations are treated as ground truth for video-based evaluation, reconstruction and annotation artifacts can miss objects that are clearly visible in the video, mislabel object identities, or corrupt geometry-dependent answers (e.g., size), yielding incorrect or ambiguous QA pairs. Second, evaluations often assume full-scene access, while many VLMs operate on sparsely sampled frames (e.g., 16-64), making many questions effectively unanswerable under the actual model inputs. We improve evaluation validity by introducing ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs. To this end, we re-annotate objects and geometry across 381 scenes from 5 datasets to improve data quality, and regenerate all QA pairs with rigorous bias mitigation and human verification using professional 3D annotation tools. We further enhance evaluation controllability by providing variants across multiple frame budgets (16/32/64/all) and fine-grained object visibility metadata, enabling controlled diagnostic analyses. Evaluations of general and domain-specific VLMs on ReVSI reveal systematic failure modes that are obscured by prior benchmarks, yielding a more reliable and diagnostic assessment of spatial intelligence.

Related papers

• VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal [41.67024349583551]
  We introduce VOR Bench, which advances VOR evaluation through three integrated components.<n>First, we present the VOR dataset (VORD), the first benchmark dataset providing both paired edited videos and graffiti masks.<n>Second, we develop rMPAF, a realistic Motion- capable Paired-video Acquisition Framework.<n>Third, we introduce VOR-MDSM, the first perception-driven VLM-based scoring model specifically designed for mask-guided VOR.
  arXiv  Detail & Related papers  (2026-09-15T09:08:00Z)
• IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoning [30.654858163397662]
  We introduce I-Bench, an evaluation suite that requires Vision-Language Models to predict structured 3D layouts on indoor scenes.<n>I-Bench moves beyond CLEVR-style simple geometric primitives so that any image-space discrepancy reflects spatial reasoning alone.
  arXiv  Detail & Related papers  (2026-07-03T22:04:42Z)
• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning [24.251128353444713]
  Vision-Language Models (VLMs) often struggle with robust 3D spatial reasoning.<n>We propose GASP (GeometricAware Spatial Priors), a framework that injects fundamental geometric priors directly into the LLM's transformer layers.<n>GASP employs a small correspondence head, applied as a deep supervision signal across all layers, and is trained with a dual objective leveraging ground-scale video scenes.
  arXiv  Detail & Related papers  (2026-05-28T17:00:52Z)
• Benchmarking Deflection and Hallucination in Large Vision-Language Models [25.176271096443482]
  Existing benchmarks overlook conflicts between visual and textual evidence.<n>We introduce VLM-DeflectionBench, a benchmark of 2,775 samples spanning diverse multimodal retrieval settings.<n>Our results highlight the need to evaluate not only what models know, but how they behave when they do not.
  arXiv  Detail & Related papers  (2026-04-13T20:22:22Z)
• Fix Before Search: Benchmarking Agentic Query Visual Pre-processing in Multimodal Retrieval-augmented Generation [47.96044455071274]
  V-QPP-Bench is the first benchmark dedicated to Visual Query Pre-processing.<n>Visual imperfections severely degrade both retrieval recall and end-to-end MRAG performance.<n>Off-the-shelf MLLMs struggle with tool selection and parameter prediction without specialized training.<n> supervised fine-tuning enables compact models to achieve comparable or superior performance to larger proprietary models.
  arXiv  Detail & Related papers  (2026-02-13T18:39:48Z)
• NovisVQ: A Streaming Convolutional Neural Network for No-Reference Opinion-Unaware Frame Quality Assessment [39.76658525158528]
  Video quality assessment (VQA) is vital for computer vision tasks, but existing approaches face major limitations.<n>We present a scalable, streaming-based VQA model that is both no-reference and opinion-unaware.
  arXiv  Detail & Related papers  (2025-11-06T18:23:55Z)
• EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing [170.71134330650796]
  EdiVal-Agent is an object-centric evaluation framework for instruction-based image editing.<n>It is designed to assess not only standard single-turn but also multi-turn instruction-based editing with precision.<n>We build EdiVal-Bench, a benchmark covering 9 instruction types and 13 state-of-the-art editing models spanning in-context, flow-matching, and diffusion paradigms.
  arXiv  Detail & Related papers  (2025-09-16T17:45:39Z)
• RT-VLM: Re-Thinking Vision Language Model with 4-Clues for Real-World Object Recognition Robustness [2.9979091009694088]
  Real world deployments often expose modern object recognition models to domain shifts that precipitate a severe drop in accuracy.<n>To mitigate this degradation, we introduce the Re-Thinking Vision Language Model (RT-VLM) framework.
  arXiv  Detail & Related papers  (2025-09-01T02:13:00Z)
• BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Models [2.526146573337397]
  We propose a new evaluation methodology, inspired by ophthalmologic diagnostics.<n>We use procedural generation of synthetic images to obtain control over visual attributes.<n>This diagnostic allows systematic stress testing and fine-grained failure analysis.
  arXiv  Detail & Related papers  (2025-06-05T12:43:10Z)
• E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models [78.1674905950243]
  We present the first comprehensive benchmark for 3D geometric foundation models (GFMs)<n>GFMs directly predict dense 3D representations in a single feed-forward pass, eliminating the need for slow or unavailable precomputed camera parameters.<n>We evaluate 16 state-of-the-art GFMs, revealing their strengths and limitations across tasks and domains.<n>All code, evaluation scripts, and processed data will be publicly released to accelerate research in 3D spatial intelligence.
  arXiv  Detail & Related papers  (2025-06-02T17:53:09Z)
• ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models [68.46716645478661]
  Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and reasoning about visual content.<n>Current VLMs excel primarily at egocentric spatial reasoning (from the camera's perspective) but fail to generalize to allocentric viewpoints.<n>We introduce ViewSpatial-Bench, the first comprehensive benchmark designed specifically for multi-viewpoint spatial localization recognition evaluation.
  arXiv  Detail & Related papers  (2025-05-27T17:59:26Z)
• Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models [77.96693360763925]
  Video SimpleQA is the first comprehensive benchmark tailored for factuality evaluation in video contexts.<n>Our work differs from existing video benchmarks through the following key features: Knowledge required: demanding integration of external knowledge beyond the video's explicit narrative.<n>Short-form definitive answer: Answers are crafted as unambiguous and definitively correct in a short format with minimal scoring variance.
  arXiv  Detail & Related papers  (2025-03-24T17:46:09Z)
• CONVIQT: Contrastive Video Quality Estimator [63.749184706461826]
  Perceptual video quality assessment (VQA) is an integral component of many streaming and video sharing platforms. Here we consider the problem of learning perceptually relevant video quality representations in a self-supervised manner. Our results indicate that compelling representations with perceptual bearing can be obtained using self-supervised learning.
  arXiv  Detail & Related papers  (2022-06-29T15:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.