What's Holding Back Latent Visual Reasoning?

• URL: http://arxiv.org/abs/2605.18445v2
• Date: Tue, 19 May 2026 10:08:18 GMT
• Title: What's Holding Back Latent Visual Reasoning?
• Abstract summary: We investigate how recent models leverage continuous latent tokens as intermediate visual imagination steps.<n>Surprisingly, we find that model accuracy is unaffected when latent tokens are replaced by uninformative dummy tokens.<n>Our experiments reveal two crucial issues holding back latent visual reasoning.
• Score: 23.63938540988447
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans can approach complex visual problems by mentally simulating intermediate visual steps, rather than reasoning through language alone. Inspired by this, several works on Vision-Language Models have recently explored chain-of-thought reasoning with continuous latent tokens as intermediate visual imagination steps. In this work, we investigate how recent models leverage such latent tokens. Surprisingly, we find that model accuracy is unaffected when latent tokens are replaced by uninformative dummy tokens. This indicates that latent tokens play a minimal causal role in the model's final prediction. To better understand this phenomenon, we analyze both the training signal provided by oracle latent representations and the quality of the latent tokens generated at inference time. Our experiments reveal two crucial issues holding back latent visual reasoning: First, in most existing datasets, oracle latent tokens provide limited additional information beyond the original image and do not substantially simplify the task, leading models to ignore them during training and effectively bypassing them at inference time. When fine-tuned on a diagnostic dataset, in which latent tokens provide sufficient support for the final prediction, we show that models can causally rely on them. Second, the latent tokens produced at inference time deviate from their corresponding oracle representations, collapsing to a narrow region and preventing benefits even when the model relies on them. Overall, our findings suggest that future progress in latent visual reasoning depends on two key pillars: high-quality datasets with informative intermediate steps and more precise latent token prediction.

Related papers

• Unveiling the Visual Counting Bottleneck in Vision-Language Models [49.591496870141846]
  This work deconstructs visual counting into three cognitive stages: visual individuation, magnitude awareness, and symbolic mapping.<n>Using synthetic Go boards and linear probes, we demonstrate that visual backbones maintain robust, linearly separable representations of quantity well into the extrapolation regime.<n>We pinpoint the collapse to the symbolic mapping stage, where the model fails to project valid visual magnitudes onto symbolic tokens.
  arXiv  Detail & Related papers  (2026-05-28T16:20:29Z)
• Leveraging Latent Visual Reasoning in Silence [46.71750408786006]
  We show that replacing latent tokens with random noise or removing them completely causes little performance degradation across spatial reasoning benchmarks.<n>We propose an attention-based reward that encourages generated latent tokens to interact with later text tokens during RL.
  arXiv  Detail & Related papers  (2026-05-18T16:46:02Z)
• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs [54.16324124242172]
  Continuous latent-space reasoning offers a compact alternative to textual chain-of-thought for multimodal models.<n>We identify a previously overlooked optimization pathology in existing latent visual reasoning methods.<n>We show that inference-time latent optimization, without any parameter updates, effectively unleashes the suppressed reasoning capacity of visual latents.
  arXiv  Detail & Related papers  (2026-05-04T15:36:12Z)
• Thinking into the Future: Latent Lookahead Training for Transformers [34.73973224120233]
  Autoregressive language models trained with next-token prediction generate text by sampling one discrete token at a time.<n>We introduce latent lookahead, a training strategy that enables models to "think" before generating.<n>We show that latent lookahead substantially outperforms both autoregressive and non-autoregressive baselines on planning tasks such as maze solving, Sudoku, and ProsQA.
  arXiv  Detail & Related papers  (2026-03-03T17:15:42Z)
• Imagination Helps Visual Reasoning, But Not Yet in Latent Space [65.80396132375571]
  We investigate the validity of latent reasoning using Causal Mediation Analysis.<n>We show that latent tokens encode limited visual information and exhibit high similarity.<n>We propose a straightforward alternative named CapImagine, which teaches the model to explicitly imagine using text.
  arXiv  Detail & Related papers  (2026-02-26T08:56:23Z)
• Temporal Predictors of Outcome in Reasoning Language Models [0.0]
  Chain-of-thought (CoT) paradigm uses the elicitation of step-by-step rationales as a proxy for reasoning.<n>We show that, for harder questions, a drop in predictive accuracy highlights a selection artifact.<n>Overall, our results imply that for reasoning models, internal self-assessment of success tends to emerge after only a few tokens.
  arXiv  Detail & Related papers  (2025-11-03T08:57:18Z)
• Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation [85.82112629564942]
  We propose TokenBridge, which maintains the strong representation capacity of continuous tokens while preserving the modeling simplicity of discrete tokens.<n>We introduce a dimension-wise quantization strategy that independently discretizes each feature dimension, paired with a lightweight autoregressive prediction mechanism.<n>Our approach achieves reconstruction and generation quality on par with continuous methods while using standard categorical prediction.
  arXiv  Detail & Related papers  (2025-03-20T17:59:59Z)
• I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? [76.15163242945813]
  Large language models (LLMs) have led many to conclude that they exhibit a form of intelligence.<n>We introduce a novel generative model that generates tokens on the basis of human-interpretable concepts represented as latent discrete variables.
  arXiv  Detail & Related papers  (2025-03-12T01:21:17Z)
• Causal-Inspired Multitask Learning for Video-Based Human Pose Estimation [18.826857684901118]
  We introduce a causal-temporal modeling framework consisting of two stages.<n>The first stage endows the model with causal-temporal modeling ability by introducing two self-supervision auxiliary tasks.<n>In the second stage, we argue that not all feature tokens contribute equally to pose estimation.<n>Our method outperforms state-of-the-art methods on three large-scale benchmark datasets.
  arXiv  Detail & Related papers  (2025-01-24T09:45:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.