CountSteer: Steering Attention for Object Counting in Diffusion Models

• URL: http://arxiv.org/abs/2511.11253v1
• Date: Fri, 14 Nov 2025 12:52:11 GMT
• Title: CountSteer: Steering Attention for Object Counting in Diffusion Models
• Abstract summary: Text-to-image diffusion models generate realistic and coherent images but often fail to follow numerical instructions in text.<n>We introduce CountSteer, a training-free method that improves generation of specified object counts by steering the model's cross-attention hidden states during inference.
• Score: 21.46393378362606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image diffusion models generate realistic and coherent images but often fail to follow numerical instructions in text, revealing a gap between language and visual representation. Interestingly, we found that these models are not entirely blind to numbers-they are implicitly aware of their own counting accuracy, as their internal signals shift in consistent ways depending on whether the output meets the specified count. This observation suggests that the model already encodes a latent notion of numerical correctness, which can be harnessed to guide generation more precisely. Building on this intuition, we introduce CountSteer, a training-free method that improves generation of specified object counts by steering the model's cross-attention hidden states during inference. In our experiments, CountSteer improved object-count accuracy by about 4% without compromising visual quality, demonstrating a simple yet effective step toward more controllable and semantically reliable text-to-image generation.

Related papers

• Demystifying Numerosity in Diffusion Models -- Limitations and Remedies [29.067276405073283]
  Numerosity remains a challenge for state-of-the-art text-to-image generation models like FLUX and GPT-4o.<n>Can diffusion models inherently generate the correct number of objects specified by a textual prompt simply by scaling up the dataset and model size?<n>We construct a clean synthetic numerosity benchmark comprising two complementary datasets: GrayCount250 for controlled scaling studies, and NaturalCount6 featuring complex naturalistic scenes.
  arXiv  Detail & Related papers  (2025-10-13T08:07:24Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations [83.93566096400723]
  We find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization.<n>Character-level segmentation improves string manipulation and code understanding tasks by up to +14%.<n>Right-aligned digit grouping enhances large-number arithmetic by +33%.
  arXiv  Detail & Related papers  (2025-06-23T18:02:26Z)
• CountDiffusion: Text-to-Image Synthesis with Training-Free Counting-Guidance Diffusion [82.82885671486795]
  We propose CountDiffusion, a training-free framework aiming at generating images with correct object quantity from textual descriptions.<n>The proposed CountDiffusion can be plugged into any diffusion-based text-to-image (T2I) generation models without further training.
  arXiv  Detail & Related papers  (2025-05-07T11:47:35Z)
• Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help [18.70937620674227]
  We introduce T2ICountBench, a novel benchmark designed to rigorously evaluate the counting ability of state-of-the-art text-to-image diffusion models.<n>Our evaluations reveal that all state-of-the-art diffusion models fail to generate the correct number of objects, with accuracy dropping significantly as the number of objects increases.
  arXiv  Detail & Related papers  (2025-03-10T03:28:18Z)
• T2ICount: Enhancing Cross-modal Understanding for Zero-Shot Counting [30.004769932953952]
  Zero-shot object counting aims to count instances of arbitrary object categories specified by text descriptions.<n>We present T2ICount, a diffusion-based framework that leverages rich prior knowledge and fine-grained visual understanding from pretrained diffusion models.
  arXiv  Detail & Related papers  (2025-02-28T01:09:18Z)
• Detection-Driven Object Count Optimization for Text-to-Image Diffusion Models [54.641726517633025]
  We propose a new framework that uses pre-trained object counting techniques and object detectors to guide generation.<n>First, we optimize a counting token using an outer-loop loss computed on fully generated images.<n>Second, we introduce a detection-driven scaling term that corrects errors caused by viewpoint and proportion shifts.
  arXiv  Detail & Related papers  (2024-08-21T15:51:46Z)
• Make It Count: Text-to-Image Generation with an Accurate Number of Objects [31.909039527164403]
  Control of the number of objects depicted using text is surprisingly hard. Generating object-correct counts is challenging because the generative model needs to keep a sense of separate identity for every instance of the object. We show how CountGen can be used to guide denoising with correct object count.
  arXiv  Detail & Related papers  (2024-06-14T17:46:08Z)
• Counting Guidance for High Fidelity Text-to-Image Synthesis [16.76098645308941]
  Text-to-image diffusion models sometimes struggle to create high-fidelity content for the given input prompt.<n>We present a method to improve diffusion models so that they accurately produce the correct object count based on the input prompt.
  arXiv  Detail & Related papers  (2023-06-30T11:40:35Z)
• Discriminative Class Tokens for Text-to-Image Diffusion Models [102.88033622546251]
  We propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of free-form text.<n>Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images.<n>We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier.
  arXiv  Detail & Related papers  (2023-03-30T05:25:20Z)
• Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic [72.60554897161948]
  Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences. In this work, we repurpose such models to generate a descriptive text given an image at inference time. The resulting captions are much less restrictive than those obtained by supervised captioning methods.
  arXiv  Detail & Related papers  (2021-11-29T11:01:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.