GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations

• URL: http://arxiv.org/abs/2603.10978v1
• Date: Wed, 11 Mar 2026 17:04:30 GMT
• Title: GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations
• Abstract summary: Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks.<n>We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from object detection models to mitigate hallucinations.
• Score: 19.262982037123447
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-art reasoning-capable VLMs. Conversely, CNN-based object detection models (ODMs) such as YOLO excel at spatial localization and instance counting with minimal computational overhead. We propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations. In the best case, our prompt-based augmentation strategy achieves 81.3% counting accuracy on the best-performing model (Ovis2.5-2B) - a 6.6pp improvement - while reducing inference time by 22% through elimination of hallucination-driven reasoning loops for stronger models. We conduct comprehensive ablation studies demonstrating that positional encoding is a critical component, being beneficial for stronger models but detrimental for weaker ones. Confidence scores, by contrast, introduce noise for most architectures and their removal improves performance in four of five evaluated models. We further evaluate feature-level fusion architectures, finding that explicit symbolic grounding via structured prompts outperforms implicit feature fusion despite sophisticated cross-attention mechanisms. Our approach yields consistent improvements across four of five evaluated VLM architectures (6.2--7.5pp), with one architecture exhibiting degraded performance due to incompatibility between its iterative reflection mechanisms and structured prompts. These results suggest that counting failures stem from fundamental spatial-semantic integration limitations rather than architecture-specific deficiencies, while highlighting the importance of architectural compatibility in augmentation strategies.

Related papers

• Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models [12.293070202366316]
  Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks.<n>They often exploit spurious object-background robustness, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence.<n>We introduce PURGE, a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs.
  arXiv  Detail & Related papers  (2026-08-30T19:42:47Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models [64.56065206447788]
  Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets.<n>We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometrics.<n>We show that low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions.
  arXiv  Detail & Related papers  (2026-03-06T10:58:02Z)
• Task Complexity Matters: An Empirical Study of Reasoning in LLMs for Sentiment Analysis [2.1036545320600095]
  Large language models (LLMs) with reasoning capabilities have fueled a compelling narrative that reasoning universally improves performance across language tasks.<n>We test this claim through a comprehensive evaluation of 504 configurations across seven model families.<n>Our findings reveal that reasoning effectiveness is strongly task-dependent, challenging prevailing assumptions.
  arXiv  Detail & Related papers  (2026-02-27T14:49:05Z)
• Toward More Reliable Artificial Intelligence: Reducing Hallucinations in Vision-Language Models [0.0]
  Vision-language models (VLMs) frequently generate hallucinated content plausible but incorrect claims about image content.<n>We propose a training-free self-correction framework enabling VLMs to iteratively refine responses through uncertainty-guided visual re-attention.
  arXiv  Detail & Related papers  (2025-12-08T13:58:46Z)
• An Analysis of Architectural Impact on LLM-based Abstract Visual Reasoning: A Systematic Benchmark on RAVEN-FAIR [0.0]
  GPT-4.1-Mini consistently achieved the highest overall accuracy across all architectures.<n>Each model exhibited distinct sensitivity patterns to architectural design, underscoring that reasoning effectiveness remains model-specific.
  arXiv  Detail & Related papers  (2025-11-14T22:50:22Z)
• Interpretable Few-Shot Image Classification via Prototypical Concept-Guided Mixture of LoRA Experts [79.18608192761512]
  Self-Explainable Models (SEMs) rely on Prototypical Concept Learning (PCL) to enable their visual recognition processes more interpretable.<n>We propose a Few-Shot Prototypical Concept Classification framework that mitigates two key challenges under low-data regimes: parametric imbalance and representation misalignment.<n>Our approach consistently outperforms existing SEMs by a notable margin, with 4.2%-8.7% relative gains in 5-way 5-shot classification.
  arXiv  Detail & Related papers  (2025-06-05T06:39:43Z)
• Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs [65.93003087656754]
  VisFactor is a benchmark that digitizes 20 vision-centric subtests from a well-established cognitive psychology assessment.<n>We evaluate 20 frontier Multimodal Large Language Models (MLLMs) from GPT, Gemini, Claude, LLaMA, Qwen, and SEED families.<n>The best-performing model achieves a score of only 25.19 out of 100, with consistent failures on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination.
  arXiv  Detail & Related papers  (2025-02-23T04:21:32Z)
• Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models [61.899791071654654]
  We introduce a benchmark, Q-Spatial Bench, with 271 questions across five categories designed for quantitative spatial reasoning. We investigate the performance of state-of-the-art vision-language models (VLMs) on this task. We develop a zero-shot prompting technique, SpatialPrompt, that encourages VLMs to answer quantitative spatial questions using reference objects as visual cues.
  arXiv  Detail & Related papers  (2024-09-15T16:45:42Z)
• Entity-Aware Biaffine Attention Model for Improved Constituent Parsing with Reduced Entity Violations [0.0]
  We propose an entity-aware biaffine attention model for constituent parsing. This model incorporates entity information into the biaffine attention mechanism by using additional entity role vectors for potential phrases. We introduce a new metric, the Entity Violating Rate (EVR), to quantify the extent of entity violations in parsing results.
  arXiv  Detail & Related papers  (2024-09-01T05:59:54Z)
• Towards Evaluating the Robustness of Visual State Space Models [63.14954591606638]
  Vision State Space Models (VSSMs) have demonstrated remarkable performance in visual perception tasks. However, their robustness under natural and adversarial perturbations remains a critical concern. We present a comprehensive evaluation of VSSMs' robustness under various perturbation scenarios.
  arXiv  Detail & Related papers  (2024-06-13T17:59:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.