NumBench: Diagnosing Counting Failures in Text-to-Image Models

• URL: http://arxiv.org/abs/2608.28206v1
• Date: Fri, 28 Aug 2026 11:26:19 GMT
• Title: NumBench: Diagnosing Counting Failures in Text-to-Image Models
• Abstract summary: Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why.<n>We introduce textbf, a benchmark of 640,000 prompts spanning 1,600 categories and counts from 1 to 100.<n>We also develop a process model in which requested instances compete for a finite set of resolvable image regions.
• Score: 27.391694807526857
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We introduce \textbf{NumBench}, a benchmark of 640{,}000 prompts spanning 1{,}600 categories and counts from 1 to 100. Its factorial design varies object composition, spatial guidance, and appearance conditions while balancing counts and category exposure. We also develop a process model in which requested instances compete for a finite set of resolvable image regions. The model predicts a near-quadratic collision deficit at low occupancy and shows how coordinated placement reduces it. For scalable evaluation, we propose the Confidence-Weighted Numeric Precision Score (\cwnps), which aggregates three calibrated detectors and discounts uncertain proposals. Across five commercial systems, two open models, and two specialized counting methods, performance declines sharply with requested count; all evaluated methods are weak above 50 objects. Count range has the largest measured effect, followed by layout and composition. Grid guidance is strongest among guided layouts, consistent with the coordination prediction, although the analysis does not establish collision as the sole cause. A 14{,}400-image human study supports automated evaluation through count 50, while results on 243 natural-language prompts show transfer beyond NumBench templates.

Related papers

• SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models [9.496555964908188]
  Multimodal large language models (MLLMs) make grounded predictions in real-world scenes by combining visual and textual cues.<n>We introduce SIGNPOST-Bench, a controlled counterfactual benchmark for evaluating text-vision conflict resolution.
  arXiv  Detail & Related papers  (2026-08-04T21:55:47Z)
• PaintBench: Deterministic Evaluation of Precise Visual Editing [50.50559031686293]
  PaintBench is a benchmark for 20 fundamental precise visual editing operations.<n>Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1%.<n>Fine-grained benchmark diagnostics show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size.
  arXiv  Detail & Related papers  (2026-05-29T16:01:14Z)
• Measuring Representation Robustness in Large Language Models for Geometry [7.743292557234699]
  In geometry, identical problems can be expressed in Euclidean, coordinate, or vector forms.<n>Existing benchmarks report accuracy on fixed formats.<n>We propose GeoRepEval, a representation-aware evaluation framework.
  arXiv  Detail & Related papers  (2026-04-03T11:36:49Z)
• Learning More from Less: Unlocking Internal Representations for Benchmark Compression [37.69575776639016]
  We introduce REPCORE, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets.<n>Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy.
  arXiv  Detail & Related papers  (2026-01-31T13:11:39Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• Beyond 'Templates': Category-Agnostic Object Pose, Size, and Shape Estimation from a Single View [69.6117755984012]
  Estimating an object's 6D pose, size, and shape from visual input is a fundamental problem in computer vision.<n>We propose a unified category-agnostic framework that simultaneously predicts 6D pose, size, and dense shape from a single RGB-D image.
  arXiv  Detail & Related papers  (2025-10-13T17:49:15Z)
• The NazoNazo Benchmark: A Cost-Effective and Extensible Test of Insight-Based Reasoning in LLMs [3.9977256267361754]
  We present Nazonazo, a cost-effective benchmark built from Japanese children's riddles to test insight-based reasoning.<n>No model except for GPT-5 is comparable to human performance, which achieves a 52.9% mean accuracy.
  arXiv  Detail & Related papers  (2025-09-18T07:50:04Z)
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling [81.34900892130929]
  We explore inference compute as another axis for scaling, using the simple technique of repeatedly sampling candidate solutions from a model.<n>Across multiple tasks and models, we observe that coverage scales with the number of samples over four orders of magnitude.<n>In domains like coding and formal proofs, where answers can be automatically verified, these increases in coverage directly translate into improved performance.
  arXiv  Detail & Related papers  (2024-07-31T17:57:25Z)
• Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence [80.6840060272386]
  This paper identifies the importance of being geometry-aware for semantic correspondence. We show that incorporating this information can markedly enhance semantic correspondence performance. Our method achieves a PCK@0.10 score of 65.4 (zero-shot) and 85.6 (supervised) on the challenging SPair-71k dataset.
  arXiv  Detail & Related papers  (2023-11-28T18:45:13Z)
• Single-Stage Visual Relationship Learning using Conditional Queries [60.90880759475021]
  TraCQ is a new formulation for scene graph generation that avoids the multi-task learning problem and the entity pair distribution. We employ a DETR-based encoder-decoder conditional queries to significantly reduce the entity label space as well. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of-the-art two-stage methods on the Visual Genome dataset.
  arXiv  Detail & Related papers  (2023-06-09T06:02:01Z)
• CounTR: Transformer-based Generalised Visual Counting [94.54725247039441]
  We develop a computational model for counting the number of objects from arbitrary semantic categories, using arbitrary number of "exemplars" We conduct thorough ablation studies on the large-scale counting benchmark, e.g. FSC-147, and demonstrate state-of-the-art performance on both zero and few-shot settings.
  arXiv  Detail & Related papers  (2022-08-29T17:02:45Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.