CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models

• URL: http://arxiv.org/abs/2605.17826v1
• Date: Mon, 18 May 2026 04:00:05 GMT
• Title: CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models
• Abstract summary: Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors.<n>We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs.<n>We show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens.
• Score: 44.585250047959924
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors. Counting provides a precise testbed: when visual evidence conflicts with canonical object knowledge, a model must rely on the image rather than a prototypical count. We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs, consisting of paired factual and counterfactual images with edited count-relevant attributes, verified answers, and localized evidence annotations. Evaluating recent VLMs, we find strong performance on factual images but consistent degradation under counterfactual attribute changes, indicating reliance on object-level priors even when contradictory visual evidence is present. Using localized annotations, we show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens. We introduce a unified inference-time attention modulation strategy that reweights selected visual tokens, improving counterfactual counting accuracy by up to 8% across multiple VLMs. Overall, CounterCount exposes prior-driven counting failures and provides diagnostic insights for designing future VLMs.

Related papers

• MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts [10.608824155356377]
  We present MissingBench-Verified, a benchmark designed to evaluate a specific and practically relevant scenario.<n>We observe consistent and significant failure rates that persist even when external tool evidence explicitly contradicts the model's visual perception.<n>We find that existing mitigation strategies, including tool-assisted verification, autonomous visual reasoning, longer reasoning durations, and fine-tuning on an easier dataset, provide negligible improvement.
  arXiv  Detail & Related papers  (2026-07-21T03:43:50Z)
• Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs [14.028133377242701]
  Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself.<n>We introduce PriVE-Bench, a Prior-vs-Visual Evidence Benchmark that uses paired original and counterfactual images to distinguish visually grounded answers from prior-consistent errors.<n>We also introduce PriVE-Tools, a controlled agentic-vision-inspired extension that evaluates whether tool-derived visual evidence improves grounding under the same counterfactual conflicts.
  arXiv  Detail & Related papers  (2026-07-14T16:45:02Z)
• Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy [27.323597665236644]
  Large Vision-Language Models (LVLMs) show strong visualization interpretation.<n>It is unclear whether their responses reflect genuine reasoning over visual evidence or factual priors learned during training.<n>We present a framework that isolates visual correctness from factual correctness.
  arXiv  Detail & Related papers  (2026-06-02T04:36:32Z)
• Unveiling the Visual Counting Bottleneck in Vision-Language Models [49.591496870141846]
  This work deconstructs visual counting into three cognitive stages: visual individuation, magnitude awareness, and symbolic mapping.<n>Using synthetic Go boards and linear probes, we demonstrate that visual backbones maintain robust, linearly separable representations of quantity well into the extrapolation regime.<n>We pinpoint the collapse to the symbolic mapping stage, where the model fails to project valid visual magnitudes onto symbolic tokens.
  arXiv  Detail & Related papers  (2026-05-28T16:20:29Z)
• Count Anything at Any Granularity [55.70950689938894]
  KubriCount is the largest and most comprehensively annotated counting dataset to date.<n>We train a multi-grained counting model that jointly leverages text and visual exemplars as complementary target specifications.<n>HieraCount substantially improves multi-grained counting accuracy and generalizes robustly to challenging real-world scenarios.
  arXiv  Detail & Related papers  (2026-05-11T17:32:37Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• Counting to Four is still a Chore for VLMs [3.017198998175514]
  Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks.<n>They still fail on simple grounding skills such as object counting.<n>We present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis.
  arXiv  Detail & Related papers  (2026-04-11T05:23:19Z)
• Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification [55.357038267439684]
  Visual Re-Examination (VRE) is a self-evolving training framework that enables MLLMs to autonomously perform visual introspection during reasoning without additional visual inputs.<n>VRE consistently improves reasoning accuracy and perceptual reliability, while substantially reducing hallucinations, especially in long-chain settings.
  arXiv  Detail & Related papers  (2026-03-27T12:22:13Z)
• When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations [29.03749623289173]
  We evaluate Vision Language Models (VLMs) on misleading visualization-caption pairs grounded in a fine-grained taxonomy of reasoning errors.<n>We develop a benchmark that combines real-world visualization with human-authored, curated misleading captions.
  arXiv  Detail & Related papers  (2026-03-23T05:34:47Z)
• Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Models [8.630726904040781]
  Explicit Visual Premise Verification (EVPV) is a lightweight verification interface that conditions step scoring on the reliability of the visual premises a step depends on.<n>EVPV improves step-level verification and consistently boosts Best-of-N reranking accuracy over strong baselines.
  arXiv  Detail & Related papers  (2026-03-17T08:40:26Z)
• Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs [60.93949629734977]
  We propose Visual Contrastive Self-Taught Reasoner (VC-STaR) to mitigate hallucinations in model-generated rationales.<n>We collect a diverse suite of VQA datasets, curate contrastive pairs according to multi-modal similarity, and generate rationales using VC-STaR.<n>Extensive experiments show that VC-STaR not only outperforms existing self-improving approaches but also surpasses models finetuned on the SoTA visual reasoning datasets.
  arXiv  Detail & Related papers  (2026-03-03T03:18:31Z)
• Beyond Accuracy: Evaluating Grounded Visual Evidence in Thinking with Images [34.324634481264034]
  We propose ViEBench, a process-verifiable benchmark designed to evaluate faithful visual reasoning.<n>Comprising 200 multi-scenario high-resolution images with expert-annotated visual evidence, ViEBench categorizes tasks by difficulty into perception and reasoning dimensions.<n>Our experiments yield several interesting observations: (1) VLMs can sometimes produce correct final answers despite grounding on irrelevant regions, and (2) they may successfully locate the correct evidence but still fail to utilize it to reach accurate conclusions.
  arXiv  Detail & Related papers  (2026-01-14T07:25:15Z)
• BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Models [2.526146573337397]
  We propose a new evaluation methodology, inspired by ophthalmologic diagnostics.<n>We use procedural generation of synthetic images to obtain control over visual attributes.<n>This diagnostic allows systematic stress testing and fine-grained failure analysis.
  arXiv  Detail & Related papers  (2025-06-05T12:43:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.