Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation

• URL: http://arxiv.org/abs/2603.19158v1
• Date: Thu, 19 Mar 2026 17:12:03 GMT
• Title: Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation
• Abstract summary: We introduce Adaptive Auxiliary Prompt Blending (AAPB) to stabilize the diffusion process in low-density regions.<n>AAPB provides semantic support in rare concept generation and structural support in image editing.<n>We empirically show consistent improvements on the RareBench and FlowEdit datasets.
• Score: 6.826122099204317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion-based text-to-image (T2I) models have made remarkable progress in generating photorealistic and semantically rich images. However, when the target concepts lie in low-density regions of the training distribution, these models often produce semantically misaligned or structurally inconsistent results. This limitation arises from the long-tailed nature of text-image datasets, where rare concepts or editing instructions are underrepresented. To address this, we introduce Adaptive Auxiliary Prompt Blending (AAPB) - a unified framework that stabilizes the diffusion process in low-density regions. AAPB leverages auxiliary anchor prompts to provide semantic support in rare concept generation and structural support in image editing, ensuring faithful guidance toward the target prompt. Unlike prior heuristic prompt alternation methods, AAPB derives a closed-form adaptive coefficient that optimally balances the influence between the auxiliary anchor and the target prompt at each diffusion step. Grounded in Tweedie's identity, our formulation provides a principled and training-free framework for adaptive prompt blending, ensuring stable and target-faithful generation. We demonstrate the effectiveness of adaptive interpolation over fixed interpolation through controlled experiments and empirically show consistent improvements on the RareBench and FlowEdit datasets, achieving superior semantic accuracy and structural fidelity compared to prior training-free baselines.

Related papers

• Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization [15.96706698967133]
  Text Embedding Steering (TES) is a training-free framework that mitigates demographic bias by directly optimizing conditional text embeddings during the diffusion process.<n>We show that a two-stage strategy - early-stage global alignment followed by iterative denoising-time refinement with CLIP-based feedback - enables stable and controllable attribute steering without modifying model parameters.
  arXiv  Detail & Related papers  (2026-07-01T11:41:40Z)
• Improving Visual Representation Alignment Generation with GRPO [51.071351994330605]
  diffusion transformers have demonstrated strong image synthesis capabilities but remain inefficient to train due to weak alignment.<n>We propose VRPO, a reinforcement-based optimization strategy that replaces REPA's static alignment loss with a generative representation policy optimization objective.<n>Our VRPO-driven training seamlessly integrates into diffusion transformers, introducing negligible cost and preserving full compatibility with SiT and DiT architectures.
  arXiv  Detail & Related papers  (2026-05-30T07:21:40Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation [65.068801413044]
  Continual test-time adaptation adapts a source-pretrained model to non-stationary, unlabeled target streams.<n>We introduce MoASE, a mixture-of-experts that disentangles domain-agnostic structure from domain-specific texture.
  arXiv  Detail & Related papers  (2026-05-18T01:52:12Z)
• ACPO: Anchor-Constrained Perceptual Optimization for Diffusion Models with No-Reference Quality Guidance [36.4379549994663]
  We investigate the problem of incorporating no-reference perceptual quality into diffusion training.<n>A key challenge is that directly optimizing perceptual signals, such as those provided by no-reference image quality assessment (NR-IQA) models, introduces a mismatch with the original diffusion objective.<n>We propose an anchor-constrained optimization framework that enables stable perceptual adaptation.
  arXiv  Detail & Related papers  (2026-04-29T07:00:00Z)
• Coevolving Representations in Joint Image-Feature Diffusion [13.292849179066993]
  We argue that the representation space guiding diffusion should itself adapt to the generative task.<n>We propose Coevolving Diffusion Representation (CoReDi), a framework in which the semantic representation space evolves during training.<n>Experiments show that CoReDi achieves faster convergence and higher sample quality compared to joint diffusion models operating in fixed representation spaces.
  arXiv  Detail & Related papers  (2026-04-19T15:29:15Z)
• Training-Free Representation Guidance for Diffusion Models with a Representation Alignment Projector [14.027059904924135]
  We introduce a representation alignment projector that injects representations predicted by a projector into intermediate sampling steps.<n>Experiments on SiTs and REPAs show notable improvements in class-conditional ImageNet synthesis.<n>The proposed method outperforms representative guidance when applied to SiT models.
  arXiv  Detail & Related papers  (2026-01-30T02:29:54Z)
• Fast, faithful and photorealistic diffusion-based image super-resolution with enhanced Flow Map models [8.773008452897791]
  FlowMapSR is a novel diffusion-based framework for image super-resolution explicitly designed for efficient inference.<n>We show that FlowMapSR achieves a better balance between reconstruction faithfulness and photorealism than recent state-of-the-art methods for both x4 and x8 upscaling.
  arXiv  Detail & Related papers  (2026-01-23T11:25:04Z)
• Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers [55.15722080205737]
  Edit2Perceive is a unified diffusion framework that adapts editing models for depth, normal, and matting.<n>Our single-step deterministic inference yields up to faster runtime while training on relatively small datasets.
  arXiv  Detail & Related papers  (2025-11-24T01:13:51Z)
• Enhancing CLIP Robustness via Cross-Modality Alignment [54.01929554563447]
  We propose Cross-modality Alignment, an optimal transport-based framework for vision-language models.<n> COLA restores global image-text alignment and local structural consistency in the feature space.<n> COLA is training-free and compatible with existing fine-tuned models.
  arXiv  Detail & Related papers  (2025-10-28T03:47:44Z)
• Dynamic Classifier-Free Diffusion Guidance via Online Feedback [53.54876309092376]
  "One-size-all" approach fails to adapt to the diverse requirements of different prompts.<n>We introduce a framework for dynamic CFG scheduling.<n>We demonstrate the effectiveness of our approach on both small-scale models and the state-of-the-art Imagen 3.
  arXiv  Detail & Related papers  (2025-09-19T16:27:19Z)
• Steering Guidance for Personalized Text-to-Image Diffusion Models [19.550718192994353]
  Existing sampling guidance methods fail to guide the output toward well-balanced space.<n>We propose personalization guidance, a simple yet effective method leveraging an unlearned weak model conditioned on a null text prompt.<n>Our method explicitly steers the outputs toward a balanced latent space without additional computational overhead.
  arXiv  Detail & Related papers  (2025-08-01T05:02:26Z)
• FreeBlend: Advancing Concept Blending with Staged Feedback-Driven Interpolation Diffusion [9.51081615960585]
  Concept blending is a promising yet underexplored area in generative models.<n>We introduce FreeBlend, an effective, training-free framework designed to address these challenges.
  arXiv  Detail & Related papers  (2025-02-08T15:25:03Z)
• Diffusion-Based Conditional Image Editing through Optimized Inference with Guidance [46.922018440110826]
  We present a training-free approach for text-driven image-to-image translation based on a pretrained text-to-image diffusion model.<n>Our method achieves outstanding image-to-image translation performance on various tasks when combined with the pretrained Stable Diffusion model.
  arXiv  Detail & Related papers  (2024-12-20T11:15:31Z)
• TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimization [59.412236435627094]
  TALE is a training-free framework harnessing the generative capabilities of text-to-image diffusion models. We equip TALE with two mechanisms dubbed Adaptive Latent Manipulation and Energy-guided Latent Optimization. Our experiments demonstrate that TALE surpasses prior baselines and attains state-of-the-art performance in image-guided composition.
  arXiv  Detail & Related papers  (2024-08-07T08:52:21Z)
• Inter-class Discrepancy Alignment for Face Recognition [55.578063356210144]
  We propose a unified framework calledInter-class DiscrepancyAlignment(IDA) IDA-DAO is used to align the similarity scores considering the discrepancy between the images and its neighbors. IDA-SSE can provide convincing inter-class neighbors by introducing virtual candidate images generated with GAN.
  arXiv  Detail & Related papers  (2021-03-02T08:20:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.