Counting to Four is still a Chore for VLMs

• URL: http://arxiv.org/abs/2604.10039v1
• Date: Sat, 11 Apr 2026 05:23:19 GMT
• Title: Counting to Four is still a Chore for VLMs
• Abstract summary: Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks.<n>They still fail on simple grounding skills such as object counting.<n>We present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis.
• Score: 3.017198998175514
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.

Related papers

• Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds [12.841957617988148]
  We study how the visual presentation of a task shapes model performance and failure modes when the underlying reasoning problem is unchanged.<n>We introduce lightweight input-side scaffolds that preserve the visual modality while making spatial structure more accessible.<n>We find that visual presentation is a central factor that determines whether VLM benchmarks measure grounded perception, downstream reasoning, or a mixture of both.
  arXiv  Detail & Related papers  (2026-08-21T14:40:26Z)
• Diagnosing Visual Ignorance in Vision-Language Models [29.2851901986069]
  Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.<n>In this work, we study language-prior reliance from both mechanistic and behavioral perspectives.<n>We find that language-prior reliance is a systematic routing failure affecting both model internals and benchmark validity.
  arXiv  Detail & Related papers  (2026-06-05T04:16:01Z)
• Unveiling the Visual Counting Bottleneck in Vision-Language Models [49.591496870141846]
  This work deconstructs visual counting into three cognitive stages: visual individuation, magnitude awareness, and symbolic mapping.<n>Using synthetic Go boards and linear probes, we demonstrate that visual backbones maintain robust, linearly separable representations of quantity well into the extrapolation regime.<n>We pinpoint the collapse to the symbolic mapping stage, where the model fails to project valid visual magnitudes onto symbolic tokens.
  arXiv  Detail & Related papers  (2026-05-28T16:20:29Z)
• Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision? [10.315515647818009]
  Benchmark accuracy is often assumed to reflect grounded visual understanding in vision-language models.<n>We systematically investigate this mismatch in a set of open-source vision-language models.<n>We find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence.
  arXiv  Detail & Related papers  (2026-05-21T17:35:04Z)
• CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models [44.585250047959924]
  Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors.<n>We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs.<n>We show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens.
  arXiv  Detail & Related papers  (2026-05-18T04:00:05Z)
• Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models [154.39583176906893]
  Large language models (MLs) have advanced understanding, they remain highly prone to hallucinations in scenes.<n>We argue this stems from a failure intemporal monitoring, the ability to persistently track object dynamic identities, states, and relations over time.<n>Existing benchmarks obscure this deficit relying on single final-answer queries that can often be resolved via local visual cues or statistical priors.
  arXiv  Detail & Related papers  (2026-05-09T14:32:36Z)
• Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification [55.357038267439684]
  Visual Re-Examination (VRE) is a self-evolving training framework that enables MLLMs to autonomously perform visual introspection during reasoning without additional visual inputs.<n>VRE consistently improves reasoning accuracy and perceptual reliability, while substantially reducing hallucinations, especially in long-chain settings.
  arXiv  Detail & Related papers  (2026-03-27T12:22:13Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs [7.406217790017003]
  We introduce VisRes Bench, a benchmark to study visual reasoning in naturalistic settings without contextual language supervision.<n>Analyzing model behavior across three levels of complexity, we uncover clear limitations in perceptual and relational visual reasoning capacities.<n>We conclude by discussing how VisRes provides a unified framework for advancing abstract visual reasoning in multimodal research.
  arXiv  Detail & Related papers  (2025-12-24T14:18:38Z)
• [De|Re]constructing VLMs' Reasoning in Counting [2.1856941852799134]
  We study the reasoning skills of seven state-of-the-art Vision-Language Models (VLMs) in the counting task under controlled experimental conditions.<n>A layer-wise analysis reveals that errors are due to incorrect mapping of the last-layer representation into the output space.<n>Our targeted training shows that fine-tuning just the output layer improves accuracy by up to 21%.
  arXiv  Detail & Related papers  (2025-10-22T13:08:47Z)
• Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting [16.47788191300739]
  Vision-Language Models (VLMs) have become a central focus of today's AI community.<n>This paper introduces a benchmark, VLMCountBench, designed under a minimalist setting with only basic geometric shapes.<n>We study the effects of simple properties such as color, size, and prompt refinement in a controlled ablation.
  arXiv  Detail & Related papers  (2025-10-06T00:11:24Z)
• SpatialViz-Bench: An MLLM Benchmark for Spatial Visualization [44.427830927596204]
  SpatialViz-Bench is a comprehensive benchmark for spatial visualization with 12 tasks across 4 sub-abilities, comprising 1,180 automatically generated problems.<n>Our evaluation of 33 state-of-the-art MLLMs reveals wide performance variations and uncovers counter-intuitive findings.
  arXiv  Detail & Related papers  (2025-07-10T10:27:20Z)
• Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios [69.00444996464662]
  We propose RIV-CoT, a Retrieval-Based Interleaved Visual Chain-of-Thought method that enables vision-language models to reason using visual crops corresponding to relevant entities.<n>Our experiments demonstrate that RIV-CoT improves answer accuracy by 3.1% and reasoning accuracy by 4.6% over vanilla CoT prompting.
  arXiv  Detail & Related papers  (2025-01-08T18:31:16Z)
• VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models [57.43276586087863]
  Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs. Existing benchmarks are often limited in scope, focusing mainly on object hallucinations. We introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases.
  arXiv  Detail & Related papers  (2024-04-22T04:49:22Z)
• NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision [64.83085920775316]
  We introduce NPHardEval4V, a multimodal benchmark suite grounded in four classical NP-hard problems.<n>Each task is presented through a combination of structured visual layouts and textual prompts, designed to assess the ability of LVLMs to perform reasoning under visual-linguistic constraints.<n>Our results show that while these models perform reasonably well on perception-based inputs, they struggle with global optimization, abstraction, and constraint satisfaction.
  arXiv  Detail & Related papers  (2024-03-04T07:10:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.