On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers

• URL: http://arxiv.org/abs/2603.28762v1
• Date: Mon, 30 Mar 2026 17:59:13 GMT
• Title: On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers
• Abstract summary: Repulsion in the Contextual Space produces significantly richer diversity without sacrificing visual fidelity or semantic adherence.<n>Our results demonstrate that repulsion in the Contextual Space produces significantly richer diversity without sacrificing visual fidelity or semantic adherence.
• Score: 45.58775624703937
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern Text-to-Image (T2I) diffusion models have achieved remarkable semantic alignment, yet they often suffer from a significant lack of variety, converging on a narrow set of visual solutions for any given prompt. This typicality bias presents a challenge for creative applications that require a wide range of generative outcomes. We identify a fundamental trade-off in current approaches to diversity: modifying model inputs requires costly optimization to incorporate feedback from the generative path. In contrast, acting on spatially-committed intermediate latents tends to disrupt the forming visual structure, leading to artifacts. In this work, we propose to apply repulsion in the Contextual Space as a novel framework for achieving rich diversity in Diffusion Transformers. By intervening in the multimodal attention channels, we apply on-the-fly repulsion during the transformer's forward pass, injecting the intervention between blocks where text conditioning is enriched with emergent image structure. This allows for redirecting the guidance trajectory after it is structurally informed but before the composition is fixed. Our results demonstrate that repulsion in the Contextual Space produces significantly richer diversity without sacrificing visual fidelity or semantic adherence. Furthermore, our method is uniquely efficient, imposing a small computational overhead while remaining effective even in modern "Turbo" and distilled models where traditional trajectory-based interventions typically fail.

Related papers

• Isotropic Embedding Perturbations for Robust Vision Language Encoders [73.92626712781859]
  Aether is a simple plug-in method that applies diffusion-style random perturbations in the embedding space via controlled alpha-mixing.<n>Inspired by feature-space perturbations in language models and image degradation in generative pretraining, Aether induces mild yet effective perturbations.<n>Aether delivers consistent gains over the advanced recipe combining CutMix, Mixup, DropPath, and RandAug.
  arXiv  Detail & Related papers  (2026-09-09T15:08:05Z)
• Coevolving Representations in Joint Image-Feature Diffusion [13.292849179066993]
  We argue that the representation space guiding diffusion should itself adapt to the generative task.<n>We propose Coevolving Diffusion Representation (CoReDi), a framework in which the semantic representation space evolves during training.<n>Experiments show that CoReDi achieves faster convergence and higher sample quality compared to joint diffusion models operating in fixed representation spaces.
  arXiv  Detail & Related papers  (2026-04-19T15:29:15Z)
• Latent Bias Alignment for High-Fidelity Diffusion Inversion in Real-World Image Reconstruction and Manipulation [25.312675775144154]
  Text-to-image diffusion models are capable of generating high-quality images guided by text prompts.<n>Can they be used to generate or approximate real-world images from the seed noise?<n> diffusion inversion problem serves as building block for bridging diffusion models and real-world scenarios.
  arXiv  Detail & Related papers  (2026-03-25T03:44:21Z)
• Scaling Dense Event-Stream Pretraining from Visual Foundation Models [112.44243079477137]
  We launch a novel self-supervised pretraining method that distills visual foundation models (VFMs) to push the boundaries of event representation at scale.<n>We curate an extensive synchronized image-event collection to amplify cross-modal alignment.<n>We extend the alignment objective to semantic structures provided off-the-shelf by VFMs, indicating a broader receptive field and stronger supervision.
  arXiv  Detail & Related papers  (2026-03-04T12:06:09Z)
• Unpaired Image-to-Image Translation via a Self-Supervised Semantic Bridge [59.247871132422006]
  Adversarial diffusion and diffusion-inversion methods have advanced unpaired image-to-image translation, but each faces key limitations.<n>We propose the Self-Supervised Semantic Bridge ( SSB), a versatile framework that integrates external semantic priors into diffusion bridge models.<n>Our key idea is to leverage self-supervised visual encoders to learn representations that are invariant to appearance changes but capture geometric structure.
  arXiv  Detail & Related papers  (2026-02-18T18:05:00Z)
• Rectifying Latent Space for Generative Single-Image Reflection Removal [16.341477336909765]
  Single-image removal is a highly ill-posed problem, where existing methods struggle to reason about the composition of corrupted regions.<n>This work reframes an editing-purpose latent diffusion model to effectively perceive and process highly ambiguous, layered image inputs.
  arXiv  Detail & Related papers  (2025-12-06T09:16:14Z)
• DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models [23.12099227251494]
  We introduce Diverse VAR, a framework that enhances the diversity of text-conditioned visual autoregressive models ( VAR) at test time.<n>Var models have emerged as strong competitors to diffusion and flow models for image generation.<n>Var models suffer from a critical limitation in diversity, often producing nearly identical images even for simple prompts.
  arXiv  Detail & Related papers  (2025-11-26T14:06:52Z)
• Training-Free Generation of Diverse and High-Fidelity Images via Prompt Semantic Space Optimization [50.5332987313297]
  We propose Token-Prompt embedding Space Optimization (TPSO), a training-free and model-agnostic module.<n>TPSO introduces learnable parameters to explore underrepresented regions of the token embedding space, reducing the tendency of the model to repeatedly generate samples from strong modes of the learned distribution.<n>In experiments on MS-COCO and three diffusion backbones, TPSO significantly enhances generative diversity, improving baseline performance from 1.10 to 4.18 points, without sacrificing image quality.
  arXiv  Detail & Related papers  (2025-11-25T00:42:09Z)
• UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception [54.53657134205492]
  UniAlignment is a unified multimodal generation framework within a single diffusion transformer.<n>It incorporates both intrinsic-modal semantic alignment and cross-modal semantic alignment, thereby enhancing the model's cross-modal consistency and instruction-following robustness.<n>We present SemGen-Bench, a new benchmark specifically designed to evaluate multimodal semantic consistency under complex textual instructions.
  arXiv  Detail & Related papers  (2025-09-28T09:11:30Z)
• Fine-grained Appearance Transfer with Diffusion Models [23.29713777525402]
  Image-to-image translation (I2I) seeks to alter the visual appearance between images while maintaining structural coherence. This paper proposes an innovative framework designed to surmount these challenges by integrating various aspects of semantic matching, appearance transfer, and latent deviation.
  arXiv  Detail & Related papers  (2023-11-27T04:00:04Z)
• Auto-regressive Image Synthesis with Integrated Quantization [55.51231796778219]
  This paper presents a versatile framework for conditional image generation. It incorporates the inductive bias of CNNs and powerful sequence modeling of auto-regression. Our method achieves superior diverse image generation performance as compared with the state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-21T22:19:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.