Unveiling the Visual Counting Bottleneck in Vision-Language Models

• URL: http://arxiv.org/abs/2605.30170v1
• Date: Thu, 28 May 2026 16:20:29 GMT
• Title: Unveiling the Visual Counting Bottleneck in Vision-Language Models
• Abstract summary: This work deconstructs visual counting into three cognitive stages: visual individuation, magnitude awareness, and symbolic mapping.<n>Using synthetic Go boards and linear probes, we demonstrate that visual backbones maintain robust, linearly separable representations of quantity well into the extrapolation regime.<n>We pinpoint the collapse to the symbolic mapping stage, where the model fails to project valid visual magnitudes onto symbolic tokens.
• Score: 49.591496870141846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Large Vision-Language Models (VLMs) excel at interpolation, they suffer catastrophic failures in systematic generalization, most notably in visual counting. In this work, we investigate this extrapolation bottleneck by deconstructing visual counting into three cognitive stages: visual individuation, magnitude awareness, and symbolic mapping. Using synthetic Go boards and linear probes, we demonstrate that visual backbones maintain robust, linearly separable representations of quantity well into the extrapolation regime, ruling out perceptual failure. Furthermore, models retain latent magnitude awareness, successfully performing comparative reasoning on quantities they fail to enumerate. We pinpoint the collapse to the symbolic mapping stage, where the model fails to project valid visual magnitudes onto symbolic tokens. Our findings support a frac tured magnitude hypothesis: VLMs fail to acquire a universal number space, instead learning disjoint, modality-specific statistical manifolds that prevent cross-modal grounding for unseen quantities. Validated on the state-of-the-art foundation model, our results suggest that bridging this gap requires inductive priors enforcing unified representations, as data scaling alone is insufficient.

Related papers

• Disentangling Semantic Attention from Structural Bias in the Attention Manifold [61.68044165974505]
  Multimodal Large Language Models (MLLMs) exhibit disproportionate attention toward certain semantically uninformative visual tokens.<n>We introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play intervention.
  arXiv  Detail & Related papers  (2026-07-27T05:30:19Z)
• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning [2.6641834518599303]
  Vision--language models (VLMs) often fail on abstract visual reasoning benchmarks such as Bongard problems.<n>We reformulate Bongard-LOGO as a symbolic reasoning task based on LOGO-style action programs or structured descriptions.
  arXiv  Detail & Related papers  (2026-04-23T07:03:48Z)
• The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm [1.0742675209112622]
  We argue that current Vision-Language Models do not faithfully synthesise multimodal data.<n>We propose a radical, information-theoretic departure: the Modality Translation Protocol.
  arXiv  Detail & Related papers  (2026-04-22T15:15:32Z)
• Counting to Four is still a Chore for VLMs [3.017198998175514]
  Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks.<n>They still fail on simple grounding skills such as object counting.<n>We present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis.
  arXiv  Detail & Related papers  (2026-04-11T05:23:19Z)
• Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs [47.94507630961399]
  We introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time.<n>We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error.<n> Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework.
  arXiv  Detail & Related papers  (2026-03-26T17:53:49Z)
• Imagination Helps Visual Reasoning, But Not Yet in Latent Space [65.80396132375571]
  We investigate the validity of latent reasoning using Causal Mediation Analysis.<n>We show that latent tokens encode limited visual information and exhibit high similarity.<n>We propose a straightforward alternative named CapImagine, which teaches the model to explicitly imagine using text.
  arXiv  Detail & Related papers  (2026-02-26T08:56:23Z)
• Kelix Technical Report [86.64551727600104]
  We present Kelix, a fully discrete autoregressive unified model that closes the understanding gap between discrete and continuous visual representations.<n>Recent work has explored discrete visual tokenization to enable fully autoregressive multimodal modeling.
  arXiv  Detail & Related papers  (2026-02-10T14:48:26Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies [40.03295633717008]
  We introduce VIA-Bench, a benchmark designed to probe model performance on visual illusions and anomalies.<n>We construct over 1K high-quality question-answer pairs that require nuanced visual reasoning.<n>Our findings reveal a fundamental divergence between machine and human perception, suggesting that resolving such perceptual bottlenecks is critical for the advancement of artificial general intelligence.
  arXiv  Detail & Related papers  (2026-02-02T08:48:03Z)
• Unleashing Perception-Time Scaling to Multimodal Reasoning Models [60.578179197783754]
  Recent advances in inference-time scaling have substantially enhanced the reasoning capabilities of Large Vision-Language Models (LVLMs)<n>Inspired by this success, similar strategies have been applied to multimodal reasoning, yet their impact on visual perception remains unclear.<n>We propose Perception-Time Scaling (PTS), a novel paradigm that encourages token-rich perception and decomposes complex perception problems into intermediate tractable sub-problems.
  arXiv  Detail & Related papers  (2025-10-10T03:17:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.