Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning

• URL: http://arxiv.org/abs/2606.31257v1
• Date: Tue, 30 Jun 2026 07:33:18 GMT
• Title: Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning
• Abstract summary: The standard way to read latent knowledge out of a model can systematically overstate what a vision-language model grounds in the image.<n>We show this on spatial reasoning, where the error is invisible to both probing and steering yet exposed by a one-line causal control.
• Score: 48.438568700670835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The standard way to read latent knowledge out of a model, a linear probe confirmed by a steering recovery, can systematically overstate what a vision-language model (VLM) actually grounds in the image. We show this on spatial reasoning, where the error is invisible to both probing and steering yet exposed by a one-line causal control: replacing the image with a gray blank. Probes decode the within-axis answer at 73--97% across axes, and a training-free projection lifts a near-chance axis from 59% to 79%, exactly the signature of unlocking latent knowledge. The blank-image arbiter refutes it, revealing three grounding regimes that probing conflates: an axis can be grounded (vision-dependent, correct), a prior (vision-independent, with its decode and its apparent recovery a directional default rather than perception), or, surprisingly, inverted: decodable, causally controllable, but deployed with the wrong sign, so the model scores below chance and the error requires looking. The taxonomy holds across the studied VLMs: in fourteen models spanning six language-model families and 2B--27B, horizontal is grounded, vertical is a prior, and depth is inverted, with the inversion emerging at scale within families. The decode-versus-deploy inversion replicates on seven of eight models across five families, and the minimal edit that re-deploys it varies with geometry: a training-free rotation matches a trained edit on the cleanest model, while distributed inversions need a trained low-rank edit, tracing a per-model correction-complexity spectrum. The cheap, self-calibrating arbiter cleanly separates grounded perception, inverted perception, and prior substitution; we argue it should be a default control for latent-knowledge and steering claims in VLMs.

Related papers

• Visual Grounding in Zero-Shot Vision-Language Control [0.0]
  Vision-language models (VLMs) are increasingly used as zero-shot controllers, but successful trajectories do not necessarily show that decisions are grounded in visual input.<n>We investigate this with an input-ablation battery: blind-image controls, repeated identical inputs, lane-axis reflection, non-visual baselines, and pipeline-integrity checks.<n>Across nine direct-action models, six structured local VLMs, and an exploratory VLM-MPC hierarchy, we analyse 32,874 scored calls over two embodiments and three simulators.
  arXiv  Detail & Related papers  (2026-08-06T15:21:42Z)
• Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning [40.28153252530189]
  Large vision-language models can reason over multimodal inputs by generating textual chains of thought.<n>A key capability exhibited in CoT reasoning is self-reflection: revisiting earlier decisions and correcting previous errors.<n>Existing LVLMs often fail to properly attend to visual inputs during reflection, limiting their ability to translate feedback into grounded corrections.<n>We propose a novel reinforcement learning training framework VRRL, with two components explicitly designed to elicit visually grounded self-reflection.
  arXiv  Detail & Related papers  (2026-07-02T17:53:15Z)
• Iterative Visual Thinking: Teaching Vision-Language Models Spatial Self-Correction through Visual Feedback [0.023227405857540805]
  Vision-language models (VLMs) achieve strong singleshot spatial grounding, yet lack any mechanism to observe and correct their own predictions.<n>We propose Iterative Visual Thinking (IVT), a closed-loop framework in which the model predicts a bounding box, observes the prediction rendered on the image, and iteratively refines through visual feedback.<n>All training uses only 2,400 samples on a single GPU, demonstrating that spatial self-correction is a learnable capability that can be instilled at modest scale.
  arXiv  Detail & Related papers  (2026-06-11T10:27:29Z)
• Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning [13.759856460598087]
  Emergent misalignment (EM) occurs when a model behaves dangerously outside the finetuning task.<n>Standard training signals can miss this shift, making reliable detection costly if it depends on repeated behavioral evaluation.<n>We ask whether emergent misalignment can instead be detected from internal representations during finetuning.
  arXiv  Detail & Related papers  (2026-05-31T04:28:21Z)
• Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering [2.497926557563177]
  Large vision-language models (LVLMs) often hallucinate objects that are not present in the input image.<n>Existing inference-time mitigation methods modify logits or hidden states throughout generation.<n>They lack an explicit grounding objective, intervene even when the model is already well-grounded, and use fixed correction strengths that do not adapt to the severity of grounding failure.<n>We propose BRACS, a training-free steering framework that addresses these issues through barrier-regulated adaptive closed-form steering.
  arXiv  Detail & Related papers  (2026-05-28T13:07:01Z)
• ETCHR: Editing To Clarify and Harness Reasoning [70.02956047187827]
  We introduce ETCHR (Editing To Clarify and Harness Reasoning), a question-conditioned, reasoning-aware image editor.<n>It is trained with a two-stage recipe targeted at the two gaps: Reasoning Imitation via supervised fine-tuning on edit trajectories, and Reasoning Enhancement with VLM-derived rewards for edit correctness and downstream reasoning accuracy.<n>Since the editor is decoupled, ETCHR plugs into different open- and closed-source MLLMs in a training-free manner.
  arXiv  Detail & Related papers  (2026-05-22T17:58:28Z)
• Do Vision--Language Models Understand 3D Scenes or Just Catalogue Objects? [0.0]
  6 frontier and open-weight VLMs, scored by trained annotators on 18,204 responses with no LLM-as-judge, reveal a sharp dissociation.<n>Models that plan rearrangements over visible layouts at 53--97% accuracy and rarely violate collision constraints fall to 6--45% on occlusion and below 7% on reflections.<n>White-box analysis on Qwen3-VL-8B-Thinking localises the failure to the visual-token merger.
  arXiv  Detail & Related papers  (2026-05-19T20:01:19Z)
• To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs [1.9349092719498848]
  Tri-Layer Diagnostic Framework disentangles hallucination sources via three metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score.<n>Our taxonomy reveals that 69.6% of samples exhibit Visual Sycophancy--models detect visual anomalies but hallucinate to satisfy user expectations--while zero samples show Robust Refusal.
  arXiv  Detail & Related papers  (2026-03-19T00:15:05Z)
• Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning [93.19037653970622]
  We introduce Spatial-SSRL, a self-supervised RL paradigm that derives verifiable signals directly from ordinary RGB or RGB-D images.<n>Training on our tasks substantially improves spatial reasoning while preserving general visual capabilities.<n>Our results show that simple, intrinsic supervision enables RLVR at scale and provides a practical route to stronger spatial intelligence in LVLMs.
  arXiv  Detail & Related papers  (2025-10-31T16:30:08Z)
• LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference [66.45326873274908]
  We propose a novel method, LoLep, which regresses Locally-Learned planes from a single RGB image to represent scenes accurately. Compared to MINE, our approach has an LPIPS reduction of 4.8%-9.0% and an RV reduction of 73.9%-83.5%.
  arXiv  Detail & Related papers  (2023-07-23T03:38:55Z)
• Lateral Ego-Vehicle Control without Supervision using Point Clouds [50.40632021583213]
  Existing vision based supervised approaches to lateral vehicle control are capable of directly mapping RGB images to the appropriate steering commands. This paper proposes a framework for training a more robust and scalable model for lateral vehicle control. Online experiments show that the performance of our method is superior to that of the supervised model.
  arXiv  Detail & Related papers  (2022-03-20T21:57:32Z)
• Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation [64.92152574895111]
  We propose a simple Orthogonal Jacobian Regularization (OroJaR) to encourage deep generative model to learn disentangled representations. Our method is effective in disentangled and controllable image generation, and performs favorably against the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-17T15:01:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.