Redirecting the Flow: Image Customization through Attention Distribution Shift

• URL: http://arxiv.org/abs/2606.16866v1
• Date: Mon, 15 Jun 2026 15:44:14 GMT
• Title: Redirecting the Flow: Image Customization through Attention Distribution Shift
• Abstract summary: Subject-driven image customization aims to generate images that follow textual instructions and preserve the identity of a given reference subject.<n>Existing approaches suffer from limited efficiency, misalignment between extracted reference features and the generative process, and interference from irrelevant information.<n>We propose CustomShift, a dual-branch architecture based on Stable Diffusion 3.<n> Experiments on the DreamBooth and Custom101 benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches.
• Score: 18.601789249339014
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Subject-driven image customization aims to generate images that not only follow textual instructions but also preserve the identity of a given reference subject. Existing approaches, including test-time fine-tuning, encoder-based methods, and token competition in shared attention spaces, suffer from limited efficiency, misalignment between extracted reference features and the generative process, and interference from irrelevant information. To address these limitations, we formulate the customization task as a distribution shift induced by incorporating reference images into text-to-image generation, and derive a Conditional Attention Distribution Shift formulation grounded in maximum entropy theory. Building on this formulation, we propose CustomShift, a dual-branch architecture based on Stable Diffusion 3. The Reference-Alignment Branch leverages self-attention between reference images and subject names to achieve layer-wise alignment with latent representations, while the Cross-Guidance Branch integrates textual and reference cues to guide generation. Experiments on the DreamBooth and Custom101 benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches, achieving a better balance between semantic fidelity and subject consistency.

Related papers

• Decoupled Guidance: Disentangling Subject and Context Pathways in Text-to-Image Personalization [13.554566178862745]
  Text-to-image personalization aims to generate a user-provided subject in novel scenes described by text.<n>Most existing methods encode subject identity (fidelity) and context (editability) through the same conditioning pathway.<n>We propose Decoupled Guidance (DeGu), a framework that routes subject identity and scene context through two independent guidance streams.
  arXiv  Detail & Related papers  (2026-07-01T10:48:43Z)
• UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation [65.53694602893042]
  We propose a unified visual conditioning framework that fuses ViT and VAE features before VLM encoding.<n>Experiments on two multi-reference generation benchmarks demonstrate that UniCustom consistently improves subject consistency, instruction following, and compositional fidelity.
  arXiv  Detail & Related papers  (2026-05-12T13:10:05Z)
• Re-Align: Structured Reasoning-guided Alignment for In-Context Image Generation and Editing [38.240269144736224]
  Re-Align bridges the gap between understanding and generation through structured reasoning-guided alignment.<n>In-context image generation and editing (ICGE) enables users to specify visual concepts through interleaved image-text prompts.
  arXiv  Detail & Related papers  (2026-01-08T17:13:00Z)
• The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignment [105.31858867473845]
  ImageCritic can be integrated into an agent framework to automatically detect inconsistencies and correct them with multi-round and local editing.<n>In experiments, ImageCritic can effectively resolve detail-related issues in various customized generation scenarios, providing significant improvements over existing methods.
  arXiv  Detail & Related papers  (2025-11-25T18:40:25Z)
• OmniRefiner: Reinforcement-Guided Local Diffusion Refinement [10.329465965964571]
  VAE-based latent compression discards subtle texture information, causing identity- and attribute-specific cues to vanish.<n>We introduce ourMthd, a detail-aware refinement framework that performs two consecutive stages of reference-driven correction.<n>Experiments demonstrate that ourMthd significantly improves reference alignment and fine-grained detail preservation.
  arXiv  Detail & Related papers  (2025-11-25T06:57:49Z)
• In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation [41.79836820271156]
  "In-Context Brush" is a zero-shot framework for customized subject insertion.<n>We formulate the object image and the textual prompts as cross-modal demonstrations.<n>The goal is to inpaint the target image with the subject aligning textual prompts without model tuning.
  arXiv  Detail & Related papers  (2025-05-26T17:49:10Z)
• Consistent Human Image and Video Generation with Spatially Conditioned Diffusion [82.4097906779699]
  Consistent human-centric image and video synthesis aims to generate images with new poses while preserving appearance consistency with a given reference image.<n>We frame the task as a spatially-conditioned inpainting problem, where the target image is in-painted to maintain appearance consistency with the reference.<n>This approach enables the reference features to guide the generation of pose-compliant targets within a unified denoising network.
  arXiv  Detail & Related papers  (2024-12-19T05:02:30Z)
• Enhancing Conditional Image Generation with Explainable Latent Space Manipulation [0.0]
  This paper proposes a novel approach to achieve fidelity to a reference image while adhering to conditional prompts. We analyze the cross attention maps of the cross attention layers and gradients for the denoised latent vector. Using this information, we create masks at specific timesteps during denoising to preserve subjects while seamlessly integrating the reference image features.
  arXiv  Detail & Related papers  (2024-08-29T03:12:04Z)
• Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization [56.12990759116612]
  Pick-and-Draw is a training-free semantic guidance approach to boost identity consistency and generative diversity for personalization methods. The proposed approach can be applied to any personalized diffusion models and requires as few as a single reference image.
  arXiv  Detail & Related papers  (2024-01-30T05:56:12Z)
• LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis [24.925757148750684]
  We propose a training-free approach for layout-to-image Synthesis that excels in producing high-quality images aligned with both textual prompts and layout instructions. LoCo seamlessly integrates into existing text-to-image and layout-to-image models, enhancing their performance in spatial control and addressing semantic failures observed in prior methods.
  arXiv  Detail & Related papers  (2023-11-21T04:28:12Z)
• RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment [112.45442468794658]
  We propose a two-stage coarse-to-fine semantic re-alignment method, named RealignDiff. In the coarse semantic re-alignment phase, a novel caption reward is proposed to evaluate the semantic discrepancy between the generated image caption and the given text prompt. The fine semantic re-alignment stage employs a local dense caption generation module and a re-weighting attention modulation module to refine the previously generated images from a local semantic view.
  arXiv  Detail & Related papers  (2023-05-31T06:59:21Z)
• Conditional Score Guidance for Text-Driven Image-to-Image Translation [52.73564644268749]
  We present a novel algorithm for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. Our method aims to generate a target image by selectively editing the regions of interest in a source image.
  arXiv  Detail & Related papers  (2023-05-29T10:48:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.