StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models

• URL: http://arxiv.org/abs/2608.20414v1
• Date: Sat, 15 Aug 2026 20:13:12 GMT
• Title: StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models
• Abstract summary: We introduce StateSight, a benchmark for cube-net opposite-face reasoning, occluded cube-tower counting, and 4-neighbor connected-component counting.<n>Each task family contains 300 single-image prompts with deterministic oracle labels and exact-match scoring.<n>OpenAI GPT-5.5, using the API model identifiergpt-5.5, achieved 59.3%, 33.3%, and 28.3% accuracy across the three tasks.
• Score: 0.5624504968545648
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models are increasingly used for multimodal question answering, yet their ability to reconstruct latent spatial structure from a single image remains difficult to isolate. Broad benchmarks often combine perception, optical character recognition, domain knowledge, linguistic priors, and reasoning in the same evaluation. We introduce StateSight, a procedurally generated benchmark for cube-net opposite-face reasoning, occluded cube-tower counting, and 4-neighbor connected-component counting. Each task family contains 300 single-image prompts with deterministic oracle labels and exact-match scoring. OpenAI GPT-5.5, using the API model identifier gpt-5.5, achieved 59.3%, 33.3%, and 28.3% accuracy across the three tasks, while Claude Sonnet 5 achieved 53.3%, 18.7%, and 7.3%. All final direct runs had zero format errors. A 30-participant human baseline on 60 items exceeded both models on every task, with mean accuracies of 80.8%, 68.8%, and 64.3%. Visible-derivation analysis identified recurring errors in image-state reconstruction and reasoning procedure. We also introduce StateSight-Steps, a companion dataset of 900 interleaved image-text examples and 3,600 deterministic intermediate visual states. The results show that format-valid responses can mask failures to recover the spatial structure required for verifiable visual inference.

Related papers

• ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making [30.869819258358405]
  Vision-language models (VLMs) have made rapid progress in visual perception and increasingly support real-world tasks that depend on images.<n>We introduce COMPLEXITYWORLD, a benchmark of 390 tasks across 39 domain-inspired visual worlds and 29 decision categories.
  arXiv  Detail & Related papers  (2026-08-05T10:05:41Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Do Vision--Language Models Understand 3D Scenes or Just Catalogue Objects? [0.0]
  6 frontier and open-weight VLMs, scored by trained annotators on 18,204 responses with no LLM-as-judge, reveal a sharp dissociation.<n>Models that plan rearrangements over visible layouts at 53--97% accuracy and rarely violate collision constraints fall to 6--45% on occlusion and below 7% on reflections.<n>White-box analysis on Qwen3-VL-8B-Thinking localises the failure to the visual-token merger.
  arXiv  Detail & Related papers  (2026-05-19T20:01:19Z)
• Medical thinking with multiple images [44.04557445622649]
  We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images.<n>The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case.<n>On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy.
  arXiv  Detail & Related papers  (2026-04-14T18:51:07Z)
• GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation [31.365285503503475]
  We present a framework for learning spatial reasoning using 2D boxes from standard detectors.<n>We show that when trained on GRAID data, models learn spatial reasoning concepts that generalize on over-detailed held-out types.<n>We also show that when trained on all questions types, achieve improvements on several existing benchmarks.
  arXiv  Detail & Related papers  (2025-10-25T02:07:23Z)
• Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales [61.03549470159347]
  Vision-language models (VLMs) have advanced rapidly, yet their capacity for image-grounded geolocation in open-world conditions has not been comprehensively evaluated.<n>We present EarthWhere, a comprehensive benchmark for VLM image geolocation that evaluates visual recognition, step-by-step reasoning, and evidence use.
  arXiv  Detail & Related papers  (2025-10-13T01:12:21Z)
• Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology [87.65242416688146]
  TreeBench is a diagnostic benchmark for visual grounded reasoning.<n>TreeVGR is a training paradigm to supervise localization and reasoning jointly with reinforcement learning.
  arXiv  Detail & Related papers  (2025-07-10T17:59:58Z)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models [48.99485386990197]
  We conduct a case study using a synthetic dataset solvable only through visual reasoning.<n>We then introduce ChartMuseum, a new Chart Question Answering (QA) benchmark containing 1,162 expert-annotated questions.<n>Although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct achieves only 38.5%.
  arXiv  Detail & Related papers  (2025-05-19T17:59:27Z)
• Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations [58.442103936918805]
  We show that Attention Mask Consistency produces superior visual grounding results than previous methods. AMC is effective, easy to implement, and is general as it can be adopted by any vision-language model.
  arXiv  Detail & Related papers  (2022-06-30T17:55:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.