Does it Really Count? Assessing Semantic Grounding in Text-Guided Class-Agnostic Counting

• URL: http://arxiv.org/abs/2605.02752v2
• Date: Wed, 13 May 2026 14:26:36 GMT
• Title: Does it Really Count? Assessing Semantic Grounding in Text-Guided Class-Agnostic Counting
• Abstract summary: Open-world text-guided class-agnostic counting (CAC) has emerged as a flexible paradigm for counting arbitrary object classes by using natural language prompts.<n>We show that several state-of-the-art CAC models often struggle to determine which object class should be counted based on the given prompt.<n>We propose a new evaluation framework focused on model robustness and trustworthiness.
• Score: 17.927293384172003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-world text-guided class-agnostic counting (CAC) has emerged as a flexible paradigm for counting arbitrary object classes by using natural language prompts. However, current evaluation protocols primarily focus on standard counting errors within single-category images, overlooking a fundamental requirement: the ability to correctly ground the textual prompt in the visual scene. In this paper, we show that several state-of-the-art CAC models often struggle to determine which object class should be counted based on the given prompt, revealing a misalignment between textual semantics and visual object representations. This limitation leads to spurious counting responses and reduced reliability in real-world scenarios. To systematically address these limitations, we propose a new evaluation framework focused on model robustness and trustworthiness. Our contribution is two-fold: (i) we introduce PrACo++ (Prompt-Aware Counting++), a novel test suite featuring two dedicated evaluation protocols -- the negative-label test and the distractor test -- paired with new specialized metrics; and (ii) we present the MUCCA (MUlti-Category Class-Agnostic counting) evaluation dataset, a new collection of real-world images featuring multiple annotated object categories per scene, unlike existing CAC benchmarks that typically include a single category per image. Our extensive experimental evaluation of 10 state-of-the-art methods shows that, despite strong performance under standard counting metrics, current models exhibit significant weaknesses in understanding and grounding object class descriptions. Finally, we provide a quantitative analysis of how semantic similarity between prompts influences these failures. Overall, our results underscore the need for more semantically grounded architectures and offer a reliable framework for future assessment in open-world text-guided CAC methods.

Related papers

• SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models [63.28435103335999]
  We introduce SOCO, a new benchmark for Semantic Object Correspondence.<n>We show that vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories.<n>We also show that LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching.
  arXiv  Detail & Related papers  (2026-05-29T17:58:48Z)
• Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection [38.31141459207419]
  We propose a novel pseudo labeling framework for open-vocabulary object detection (OVD)<n>Our approach introduces a hierarchical confidence calibration (HCC) technique, which ensures reliable class label estimation.<n>We also present LoCLIP, a parameter-efficient adaptation of CLIP that incorporates an objectness token to mitigate base class bias problem.
  arXiv  Detail & Related papers  (2026-04-25T15:06:07Z)
• Beyond Referring Expressions: Scenario Comprehension Visual Grounding [18.49037321302958]
  Existing benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category.<n>We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and context rather than explicit naming.<n>We introduce Referring Scenario (RSC), a benchmark designed for this setting. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories.
  arXiv  Detail & Related papers  (2026-04-02T17:59:08Z)
• Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inference [1.2309843977641421]
  We introduce a multi-rationale explainable object recognition benchmark comprising datasets in which each image is annotated with multiple ground-truth rationales.<n>We propose a contrastive conditional inference framework that explicitly models the probabilistic relationships among image embeddings, category labels, and rationales.<n>Our approach achieves state-of-the-art results on the multi-rationale explainable object recognition benchmark, including strong zero-shot performance.
  arXiv  Detail & Related papers  (2025-08-19T21:28:12Z)
• Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model [52.01031460230826]
  Traditional approaches rely heavily on fixed vocabularies and closed-set classification paradigms.<n>Recent research has demonstrated that combining large language models with vision-language models (VLMs) makes open-set recognition possible.<n>We propose our training-free method, Enriched-FineR, which demonstrates state-of-the-art results in fine-grained visual recognition.
  arXiv  Detail & Related papers  (2025-07-30T20:06:01Z)
• SDVPT: Semantic-Driven Visual Prompt Tuning for Open-World Object Counting [70.49268117587562]
  We propose a plug-and-play Semantic-Driven Visual Prompt Tuning framework (SDVPT) that transfers knowledge from the training set to unseen categories.<n>During inference, we dynamically synthesize the visual prompts for unseen categories based on the semantic correlation between unseen and training categories.
  arXiv  Detail & Related papers  (2025-04-24T09:31:08Z)
• Generalized Semantic Contrastive Learning via Embedding Side Information for Few-Shot Object Detection [52.490375806093745]
  The objective of few-shot object detection (FSOD) is to detect novel objects with few training samples.<n>We introduce the side information to alleviate the negative influences derived from the feature space and sample viewpoints.<n>Our model outperforms the previous state-of-the-art methods, significantly improving the ability of FSOD in most shots/splits.
  arXiv  Detail & Related papers  (2025-04-09T17:24:05Z)
• SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection [70.23196257213829]
  We propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection.<n>Our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains.<n>We then leverage large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels.
  arXiv  Detail & Related papers  (2025-03-05T09:37:05Z)
• A Survey on Class-Agnostic Counting: Advancements from Reference-Based to Open-World Text-Guided Approaches [6.356364436395916]
  We present the first comprehensive review of class-agnostic counting (CAC) methodologies.<n>We propose a taxonomy to categorize CAC approaches into three paradigms: reference-based, reference-less, and open-world text-guided.<n>We present results on the FSC-147 dataset, setting a leaderboard using gold-standard metrics, and on the CARPK dataset to assess generalization capabilities.
  arXiv  Detail & Related papers  (2025-01-31T14:47:09Z)
• Beyond Coarse-Grained Matching in Video-Text Retrieval [50.799697216533914]
  We introduce a new approach for fine-grained evaluation. Our approach can be applied to existing datasets by automatically generating hard negative test captions. Experiments on our fine-grained evaluations demonstrate that this approach enhances a model's ability to understand fine-grained differences.
  arXiv  Detail & Related papers  (2024-10-16T09:42:29Z)
• Mind the Prompt: A Novel Benchmark for Prompt-based Class-Agnostic Counting [8.000723123087473]
  Class-agnostic counting (CAC) counts instances of arbitrary object classes never seen during model training.<n>We introduce the Prompt-Aware Counting benchmark to measure the robustness and trustworthiness of prompt-based CAC models.<n>We evaluate state-of-the-art methods and demonstrate that, although some achieve impressive results on standard class-specific counting metrics, they exhibit a significant deficiency in understanding the input prompt.
  arXiv  Detail & Related papers  (2024-09-24T10:35:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.