UltraVR: A Diagnostic Ultra-Resolution Image-VQA Benchmark for Evidence-Grounded Reasoning

• URL: http://arxiv.org/abs/2606.05576v1
• Date: Thu, 04 Jun 2026 01:51:16 GMT
• Title: UltraVR: A Diagnostic Ultra-Resolution Image-VQA Benchmark for Evidence-Grounded Reasoning
• Abstract summary: We introduce UltraVR, a diagnostic benchmark for evidence-grounded visual reasoning over ultra-resolution images.<n>Each instance includes a structured ground-truth chain of thought with step-level questions, intermediate answers, and reasoning labels.<n>We show that current models remain far from reliable on ultra-resolution reasoning.
• Score: 32.37070955496229
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision-language models (VLMs) excel on visual question answering and multimodal reasoning benchmarks. Yet their capability on ultra-resolution images - where critical evidence is tiny, subtle, spatially distant, or distributed - remains unclear. Existing evaluations largely report final-answer accuracy, offering limited insight into whether models acquire and integrate the necessary visual evidence. We introduce UltraVR, a diagnostic benchmark for evidence-grounded visual reasoning over ultra-resolution images. UltraVR spans four high-value scenarios: CCTV surveillance, remote sensing (RS), whole-slide image (WSI) pathology, and industrial anomaly detection (AD). These domains pose complementary challenges: fine-grained object grounding in crowded CCTV scenes, long-range spatial comparison in RS, multi-scale evidence navigation in WSI, and subtle irregularity detection in repetitive industrial layouts. Beyond standard QA triples, each instance includes a structured ground-truth chain of thought with step-level questions, intermediate answers, and reasoning labels. These labels decompose reasoning into evidence grounding, local perception, quantification, evidence integration, and decision inference, enabling process-level diagnosis over black-box scoring. Using UltraVR, we evaluate frontier VLMs and show that current models remain far from reliable on ultra-resolution reasoning. Importantly, the structured annotations allow us to localize failures across the visual-to-decision pipeline: errors concentrate in evidence grounding and local perception, while downstream inference often recovers when intermediate visual facts are supplied. These findings demonstrate UltraVR as a diagnostic testbed for measuring not only whether VLMs answer correctly, but where their ultra-resolution reasoning process breaks.

Related papers

• VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence [26.0945130521806]
  Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation.<n>We introduce VISTAQA, a benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering.<n>We introduce GROVE, a metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean.
  arXiv  Detail & Related papers  (2026-05-20T03:44:06Z)
• UHR-Micro: Diagnosing and Mitigating the Resolution Illusion in Earth Observation VLMs [40.3198846405438]
  Vision-Language Models (VLMs) operate on ultra-high-resolution (UHR) Earth observation imagery.<n>These models are vulnerable to a severe scale mismatch between large-scale scene context and micro-scale targets.<n>We benchmark this challenge with UHR-Micro, a benchmark comprising 11,253 instructions grounded in 1,212 UHR images.
  arXiv  Detail & Related papers  (2026-05-12T15:07:30Z)
• SenseBench: A Benchmark for Remote Sensing Low-Level Visual Perception and Description in Large Vision-Language Models [9.606452752502708]
  Low-level visual perception underpins reliable remote sensing (RS) image analysis.<n>Current image quality assessment (IQA) methods output uninterpretable scalar scores rather than characterizing physics-driven RS degradations.<n>We propose SenseBench, the first dedicated diagnostic benchmark for RS low-level visual perception and description.
  arXiv  Detail & Related papers  (2026-05-11T13:48:57Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• Divide-then-Diagnose: Weaving Clinician-Inspired Contexts for Ultra-Long Capsule Endoscopy Videos [83.01543151239254]
  We define a new task, diagnosis-driven CE video summarization.<n>VideoCAP is the first CE dataset with diagnosis-driven annotations.<n>DiCE is a clinician-inspired framework that mirrors the standard CE reading workflow.
  arXiv  Detail & Related papers  (2026-04-23T16:07:51Z)
• Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI [22.439797107933355]
  We introduceSGMRI-VQA, a benchmark for multi-frame, spatially grounded reasoning on volumetric MRI.<n>Each QA pair includes a clinician-aligned chain-of-thought trace with frame-indexed bounding box coordinates.<n>We show that supervised fine-tuning of Qwen3-VL-8B with bounding box supervision consistently improves grounding performance over strong zero-shot baselines.
  arXiv  Detail & Related papers  (2026-04-17T08:06:39Z)
• RVLM: Recursive Vision-Language Models with Adaptive Depth [0.1631115063641726]
  RVLM replaces single-pass inference with an iterative generate-execute loop.<n>Every diagnostic claim is grounded in executable code, satisfying auditability requirements.<n>We evaluate on BraTS 2023 Meningioma (brain MRI) and MIMIC-CXR (chest X-ray) using Gemini 2.5 Flash without fine-tuning.
  arXiv  Detail & Related papers  (2026-03-25T11:52:55Z)
• INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs [69.11382230669491]
  hallucinations are outputs that contradict either video evidence (faithfulness) or verifiable world knowledge (factuality)<n>textscINFACT evaluates models in four modes: Base (clean), Visual Degradation, Evidence, and Temporal Intervention for order-sensitive items.<n>Experiments on 14 representative Video-LLMs reveal that higher Base-mode accuracy does not reliably translate to higher reliability in the induced modes.
  arXiv  Detail & Related papers  (2026-03-12T03:03:16Z)
• Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge [66.67024684187915]
  Large Vision Language Models (LVLMs) show immense potential for automated ophthalmic diagnosis.<n>Their clinical deployment is severely hindered by lacking domain-specific knowledge.<n>EyExIn is a framework designed to anchor retinal VLMs with expert knowledge via a Deep Expert Injection mechanism.
  arXiv  Detail & Related papers  (2026-03-07T09:43:49Z)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs [65.04549036809557]
  We introduce a benchmark built from pedestrian-perspective videos captured with synchronized stereo cameras, LiDAR, and IMU/GPS sensors.<n>This dataset provides metrically precise 3D information, enabling the automatic generation of spatial reasoning questions.<n> Evaluations reveal that the performance gains observed in structured indoor benchmarks vanish in open-world settings.
  arXiv  Detail & Related papers  (2025-12-22T18:58:12Z)
• XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography [6.447908430647854]
  We present the first systematic benchmark for evaluating cross-modal interpretability in chest X-rays.<n>We generate visual explanations using cross-attention and similarity-based localization maps.<n>We quantitatively assess their alignment with radiologist-annotated regions across multiple pathologies.
  arXiv  Detail & Related papers  (2025-10-22T13:52:19Z)
• A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection [89.92916473403108]
  This paper proposes a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework for new methods.<n>The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics.<n>We objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection.
  arXiv  Detail & Related papers  (2024-06-05T13:40:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.