HoloCount: A Holistic Visual Counting Benchmark for MLLMs

• URL: http://arxiv.org/abs/2607.06420v1
• Date: Tue, 07 Jul 2026 15:48:57 GMT
• Title: HoloCount: A Holistic Visual Counting Benchmark for MLLMs
• Abstract summary: HoloCount is a diagnostically rich benchmark structured around a three-level hierarchical taxonomy.<n>We evaluate over 20 state-of-the-art Multimodal Large Language Models (MLLMs)<n>Our findings offer a roadmap for developing more grounded and reliable multimodal systems.
• Score: 17.08907754405721
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. While Multimodal Large Language Models (MLLMs) have achieved remarkable success in qualitative scene understanding, their quantitative precision remains a significant bottleneck, often characterized by persistent numerical hallucinations. Existing counting benchmarks primarily focus on basic perception in simplified contexts, failing to capture the complex failure modes that emerge under logical constraints or adversarial conditions. To address these limitations, we introduce HoloCount, a holistic and diagnostically rich benchmark structured around a three-level hierarchical taxonomy. HoloCount evaluates MLLMs across: (1) Semantic Counting, focusing on atomic and property-based enumeration; (2) Analytical Counting, assessing logical composition through spatial and set-based reasoning; and (3) Robustness Testing, probing model integrity against adverse scenarios and grounded counter-priors, such as high-density scenes and linguistic biases. Through an exhaustive evaluation of over 20 state-of-the-art MLLMs, we reveal a critical performance gap: even top-tier models degrade significantly as tasks transition from perception to complex analytical reasoning and adverse scenarios. Our findings provide a systematic landscape of current MLLM counting capabilities and offer a roadmap for developing more grounded and reliable multimodal systems. The dataset is available at https://mm-mvr.github.io/HoloCount/.

Related papers

• Count Anything at Any Granularity [55.70950689938894]
  KubriCount is the largest and most comprehensively annotated counting dataset to date.<n>We train a multi-grained counting model that jointly leverages text and visual exemplars as complementary target specifications.<n>HieraCount substantially improves multi-grained counting accuracy and generalizes robustly to challenging real-world scenarios.
  arXiv  Detail & Related papers  (2026-05-11T17:32:37Z)
• Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios [56.703500104126014]
  DailyClue is a benchmark designed for visual clue-driven reasoning in daily scenarios.<n>Our construction is guided by two core principles: (1) strict grounding in authentic daily activities, and (2) challenging query design.<n>Our analysis reveals several critical insights, emphasizing that the accurate identification of visual clues is essential for robust reasoning.
  arXiv  Detail & Related papers  (2026-04-15T16:22:10Z)
• TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tables [13.218805579902048]
  We find that as task complexity scales, the number of involved discrete visual regions increases disproportionately.<n>This processing density leads to an internal "Perceptual Overload"<n>We introduce TableVision, a trajectory-aware benchmark designed for spatially grounded reasoning.
  arXiv  Detail & Related papers  (2026-04-04T09:26:09Z)
• Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting [59.37613121962146]
  We propose WS-COC, the first MLLM-driven weakly-supervised framework for class-agnostic object counting.<n> WS-COC matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs.
  arXiv  Detail & Related papers  (2026-02-13T09:58:35Z)
• seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs [1.0519693622157462]
  We introduce seqBench, a benchmark for probing sequential reasoning limits in Large Language Models (LLMs)<n>We find that even top-performing models systematically fail on seqBench's structured reasoning tasks despite minimal search complexity.
  arXiv  Detail & Related papers  (2025-09-21T01:32:13Z)
• CountQA: How Well Do MLLMs Count in the Wild? [1.0799568216202955]
  Multimodal Large Language Models (MLLMs) demonstrate remarkable fluency in understanding visual scenes.<n>They exhibit a critical lack in a fundamental cognitive skill: object counting.<n>This blind spot severely limits their reliability in real-world applications.<n>We introduce CountQA, a challenging new benchmark designed to probe this deficiency.
  arXiv  Detail & Related papers  (2025-08-08T04:23:04Z)
• SpatialViz-Bench: An MLLM Benchmark for Spatial Visualization [44.427830927596204]
  SpatialViz-Bench is a comprehensive benchmark for spatial visualization with 12 tasks across 4 sub-abilities, comprising 1,180 automatically generated problems.<n>Our evaluation of 33 state-of-the-art MLLMs reveals wide performance variations and uncovers counter-intuitive findings.
  arXiv  Detail & Related papers  (2025-07-10T10:27:20Z)
• SpatialScore: Towards Unified Evaluation for Multimodal Spatial Understanding [64.15606979785355]
  Multimodal large language models (MLLMs) have achieved impressive success in question-answering tasks, yet their capabilities for spatial understanding are less explored.<n>This work investigates a critical question: do existing MLLMs possess 3D spatial perception and understanding abilities?
  arXiv  Detail & Related papers  (2025-05-22T17:59:03Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.