Boosting Text-to-Image Diffusion Models via Core Token Attention-Based Seed Selection

• URL: http://arxiv.org/abs/2605.19532v1
• Date: Tue, 19 May 2026 08:33:10 GMT
• Title: Boosting Text-to-Image Diffusion Models via Core Token Attention-Based Seed Selection
• Abstract summary: We show that attention dynamics over prompt core tokens, the content-bearing words, strongly predict final generation quality.<n>We introduce Attention-Based Seed Selection (ABSS), a training-free, plug-and-play method that ranks seeds for a given prompt by leveraging cross-attention to core tokens during the denoising process.
• Score: 8.116206114186392
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-image diffusion models can synthesize high-quality images, yet the outcome is notoriously sensitive to the random seed: different initial seeds often yield large variations in image quality and prompt-image alignment. We revisit this "seed effect" and show that attention dynamics over prompt core tokens, the content-bearing words, measured during the first few denoising steps, strongly predict final generation quality. Building on this observation, we introduce Attention-Based Seed Selection (ABSS), a training-free, plug-and-play method that ranks seeds for a given prompt by leveraging cross-attention to core tokens during the denoising process. ABSS requires no finetuning and does not alter the initial noise; it scores and ranks all candidate seeds, keeps only the top-k for full generation, and discards the rest, without relying on a fixed accept/reject threshold. Operating purely at inference time, ABSS can serve as a lightweight pre-selection add-on for existing seed-optimization pipelines, enabling additional gains. Across three benchmarks, extensive experiments show that ABSS enables consistent improvements in text-image alignment and visual quality for Stable Diffusion variants, as corroborated by human preference and alignment metrics.

Related papers

• SC-Diff: Semantically Calibrated Diffusion for Visible-to-Infrared Image Translation [52.62632888060867]
  Visible-to-infrared image translation provides a practical way to expand infrared training data using abundant visible images.<n>Existing diffusion-based methods typically use semantic priors only as external conditions.<n>We propose SC-Diff, a semantically latent diffusion framework that uses semantic priors for both conditional guidance and internal self-attention calibration.
  arXiv  Detail & Related papers  (2026-08-09T07:57:40Z)
• Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time [16.04233421396159]
  AnchorSteer is a training-free framework that exerts fine-grained control over.<n>textbfAnchorSteer and its components.<n>Experiments on GenEval and T2I-CompBench++ demonstrate that AnchorSteer consistently outperforms existing baselines in text--image alignment.
  arXiv  Detail & Related papers  (2026-07-29T09:08:57Z)
• Breaking the Lock-in: Diversifying Text-to-Image Generation via Representation Modulation [25.553618756894593]
  We propose DC Attenuation for diVersity Enhancement (DAVE), a training-free representation-level intervention that selectively attenuates this component in the early regime.<n>DAVE preserves the sampling pipeline with negligible overhead, improving prompt-consistent diversity while maintaining competitive image quality.
  arXiv  Detail & Related papers  (2026-06-05T01:27:44Z)
• Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection [53.789057575175065]
  Region-Aware Prompt Augmentation (RAPTA) and Attention-Driven Multimodal Copy Detection (ADMCD) are presented.<n>RAPTA uses an object detector to find salient regions and turn them into semantically grounded prompt variants.<n>ADMCD aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation.<n>Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.
  arXiv  Detail & Related papers  (2026-03-13T15:16:27Z)
• Toward Early Quality Assessment of Text-to-Image Diffusion Models [23.306273801765297]
  Recent text-to-image (T2I) diffusion and flow-matching models can produce highly realistic images from natural language prompts.<n>Probe-Select is a plug-in module that enables efficient evaluation of image quality within the generation process.
  arXiv  Detail & Related papers  (2026-03-03T10:25:46Z)
• Noise Projection: Closing the Prompt-Agnostic Gap Behind Text-to-Image Misalignment in Diffusion Models [9.683618735282414]
  In text-to-image generation, different initial noises induce distinct denoising paths with a pretrained Stable Diffusion (SD) model.<n>While this pattern could output diverse images, some of them may fail to align well with the prompt.<n>We propose a noise projector that applies text-conditioned refinement to the initial noise before denoising.
  arXiv  Detail & Related papers  (2025-10-16T10:14:34Z)
• Early Timestep Zero-Shot Candidate Selection for Instruction-Guided Image Editing [32.56049667145546]
  ELECT (Early-timestep Latent Evaluation for Candidate Selection) is a framework that selects reliable seeds by estimating background mismatches at early diffusion timesteps.<n>It ranks seed candidates by a background inconsistency score, filtering unsuitable samples early based on background consistency while preserving editability.<n>Experiments show that ELECT reduces computational costs (by 41 percent on average) while improving background consistency and instruction adherence, achieving around 40 percent success rates in previously failed cases without any external supervision or training.
  arXiv  Detail & Related papers  (2025-04-18T05:59:01Z)
• InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization [27.508861002013358]
  InitNO is a paradigm that refines the initial noise in semantically-faithful images. A strategically crafted noise optimization pipeline is developed to guide the initial noise towards valid regions. Our method, validated through rigorous experimentation, shows a commendable proficiency in generating images in strict accordance with text prompts.
  arXiv  Detail & Related papers  (2024-04-06T14:56:59Z)
• Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Prompting [133.55037976429088]
  We investigate the adversarial robustness of vision transformers equipped with BERT pretraining (e.g., BEiT, MAE) A surprising observation is that MAE has significantly worse adversarial robustness than other BERT pretraining methods. We propose a simple yet effective way to boost the adversarial robustness of MAE.
  arXiv  Detail & Related papers  (2023-08-20T16:27:17Z)
• Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models [52.29800567587504]
  We propose a learnable sampling model, Text-Conditioned Token Selection (TCTS), to select optimal tokens via localized supervision with text information. TCTS improves not only the image quality but also the semantic alignment of the generated images with the given texts. We validate the efficacy of TCTS combined with Frequency Adaptive Sampling (FAS) with various generative tasks, demonstrating that it significantly outperforms the baselines in image-text alignment and image quality.
  arXiv  Detail & Related papers  (2023-04-04T03:52:49Z)
• Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction [138.04956118993934]
  We propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST) CST embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing.
  arXiv  Detail & Related papers  (2022-03-09T16:17:47Z)
• Fully Unsupervised Diversity Denoising with Convolutional Variational Autoencoders [81.30960319178725]
  We propose DivNoising, a denoising approach based on fully convolutional variational autoencoders (VAEs) First we introduce a principled way of formulating the unsupervised denoising problem within the VAE framework by explicitly incorporating imaging noise models into the decoder. We show that such a noise model can either be measured, bootstrapped from noisy data, or co-learned during training.
  arXiv  Detail & Related papers  (2020-06-10T21:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.