Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions

• URL: http://arxiv.org/abs/2403.17064v2
• Date: Fri, 14 Mar 2025 11:33:08 GMT
• Title: Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions
• Authors: Stefan Andreas Baumann, Felix Krause, Michael Neumayr, Nick Stracke, Melvin Sevi, Vincent Tao Hu, Björn Ommer,
• Abstract summary: Recent advances in text-to-image (T2I) diffusion models have significantly improved the quality of generated images.<n>Providing efficient control over individual subjects, particularly the attributes characterizing them, remains a key challenge.<n>No current approach offers both simultaneously, resulting in a gap when trying to achieve precise continuous and subject-specific attribute modulation.
• Score: 20.351245266660378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in text-to-image (T2I) diffusion models have significantly improved the quality of generated images. However, providing efficient control over individual subjects, particularly the attributes characterizing them, remains a key challenge. While existing methods have introduced mechanisms to modulate attribute expression, they typically provide either detailed, object-specific localization of such a modification or full-scale fine-grained, nuanced control of attributes. No current approach offers both simultaneously, resulting in a gap when trying to achieve precise continuous and subject-specific attribute modulation in image generation. In this work, we demonstrate that token-level directions exist within commonly used CLIP text embeddings that enable fine-grained, subject-specific control of high-level attributes in T2I models. We introduce two methods to identify these directions: a simple, optimization-free technique and a learning-based approach that utilizes the T2I model to characterize semantic concepts more specifically. Our methods allow the augmentation of the prompt text input, enabling fine-grained control over multiple attributes of individual subjects simultaneously, without requiring any modifications to the diffusion model itself. This approach offers a unified solution that fills the gap between global and localized control, providing competitive flexibility and precision in text-guided image generation. Project page: https://compvis.github.io/attribute-control. Code is available at https://github.com/CompVis/attribute-control.

Related papers

• Att-Adapter: A Robust and Precise Domain-Specific Multi-Attributes T2I Diffusion Adapter via Conditional Variational Autoencoder [11.392007197036525]
  We introduce the Attribute (Att) Adapter, a novel plug-and-play module designed to enable fine-grained, multi-attributes control in pretrained diffusion models. Att-Adapter is flexible, requiring no paired synthetic data for training, and is easily scalable to multiple attributes within a single model.
  arXiv  Detail & Related papers  (2025-03-15T01:06:34Z)
• MAKIMA: Tuning-free Multi-Attribute Open-domain Video Editing via Mask-Guided Attention Modulation [55.101611012677616]
  Diffusion-based text-to-image (T2I) models have demonstrated remarkable results in global video editing tasks. We present MAKIMA, a tuning-free MAE framework built upon pretrained T2I models for open-domain video editing.
  arXiv  Detail & Related papers  (2024-12-28T02:36:51Z)
• DynamicControl: Adaptive Condition Selection for Improved Text-to-Image Generation [63.63429658282696]
  We propose DynamicControl, which supports dynamic combinations of diverse control signals. We show that DynamicControl is superior to existing methods in terms of controllability, generation quality and composability under various conditional controls.
  arXiv  Detail & Related papers  (2024-12-04T11:54:57Z)
• Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theory [33.78620829249978]
  Text-to-image (T2I) diffusion models have revolutionized generative modeling by producing high-fidelity, diverse, and visually realistic images. Recent attention-based methods have improved object inclusion and linguistic binding, but still face challenges such as attribute misbinding. We propose a Bayesian approach that designs custom priors over attention distributions to enforce desirable properties. Our approach treats the attention mechanism as an interpretable component, enabling fine-grained control and improved attribute-object alignment.
  arXiv  Detail & Related papers  (2024-11-25T10:57:48Z)
• PreciseControl: Enhancing Text-To-Image Diffusion Models with Fine-Grained Attribute Control [24.569528214869113]
  StyleGAN models learn a rich face prior and enable smooth control towards fine-grained attribute editing by latent manipulation. This work uses the disentangled $mathcalW+$ space of StyleGANs to condition the T2I model. We perform extensive experiments to validate our method for face personalization and fine-grained attribute editing.
  arXiv  Detail & Related papers  (2024-07-24T07:10:25Z)
• Attribute-Aware Implicit Modality Alignment for Text Attribute Person Search [19.610244285078483]
  We propose an Attribute-Aware Implicit Modality Alignment (AIMA) framework to learn the correspondence of local representations between textual attributes and images. We show that our proposed method significantly surpasses the current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-06-06T03:34:42Z)
• FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation [99.4649330193233]
  Controllable text-to-image (T2I) diffusion models generate images conditioned on both text prompts and semantic inputs of other modalities like edge maps. We propose a novel Flexible and Efficient method, FlexEControl, for controllable T2I generation.
  arXiv  Detail & Related papers  (2024-05-08T06:09:11Z)
• Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control [58.37323932401379]
  Current diffusion models create images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. We propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.
  arXiv  Detail & Related papers  (2024-04-21T20:26:46Z)
• Box It to Bind It: Unified Layout Control and Attribute Binding in T2I Diffusion Models [28.278822620442774]
  Box-it-to-Bind-it (B2B) is a training-free approach for improving spatial control and semantic accuracy in text-to-image (T2I) diffusion models. B2B targets three key challenges in T2I: catastrophic neglect, attribute binding, and layout guidance. B2B is designed as a compatible plug-and-play module for existing T2I models.
  arXiv  Detail & Related papers  (2024-02-27T21:51:32Z)
• Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models [68.47333676663312]
  We show a simple modification of classifier-free guidance can help disentangle image factors in text-to-image models. The key idea of our method, Contrastive Guidance, is to characterize an intended factor with two prompts that differ in minimal tokens. We illustrate whose benefits in three scenarios: (1) to guide domain-specific diffusion models trained on an object class, (2) to gain continuous, rig-like controls for text-to-image generation, and (3) to improve the performance of zero-shot image editors.
  arXiv  Detail & Related papers  (2024-02-21T03:01:17Z)
• Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation [58.911255139171075]
  Controllable text generation (CTG) aims to generate text with desired attributes. We propose a novel lightweight decoding framework named Air-Decoding. Our method achieves a new state-of-the-art control performance.
  arXiv  Detail & Related papers  (2023-10-23T12:59:11Z)
• Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models [82.19740045010435]
  We introduce Uni-ControlNet, a unified framework that allows for the simultaneous utilization of different local controls and global controls. Unlike existing methods, Uni-ControlNet only requires the fine-tuning of two additional adapters upon frozen pre-trained text-to-image diffusion models. Uni-ControlNet demonstrates its superiority over existing methods in terms of controllability, generation quality and composability.
  arXiv  Detail & Related papers  (2023-05-25T17:59:58Z)
• Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models [8.250234707160793]
  Recent text-to-image (T2I) diffusion models show outstanding performance in generating high-quality images conditioned on textual prompts. They fail to semantically align the generated images with the prompts due to their limited compositional capabilities. We propose a novel attention mask control strategy based on predicted object boxes to address these issues.
  arXiv  Detail & Related papers  (2023-05-23T10:49:22Z)
• DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation [6.844825905212349]
  We propose a new CTG approach, namely DisCup, which incorporates the attribute knowledge of discriminator to optimize the control-prompts. DisCup can achieve a new state-of-the-art control performance while maintaining an efficient and high-quality text generation, only relying on around 10 virtual tokens.
  arXiv  Detail & Related papers  (2022-10-18T02:59:06Z)
• Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation [57.99007520795998]
  We discover attribute-specific control units, which consist of multiple channels of feature maps and modulation styles. Specifically, we collaboratively manipulate the modulation style channels and feature maps in control units to obtain the semantic and spatial disentangled controls. We move the modulation style along a specific sparse direction vector and replace the filter-wise styles used to compute the feature maps to manipulate these control units.
  arXiv  Detail & Related papers  (2021-11-25T10:42:10Z)
• Controllable Dialogue Generation with Disentangled Multi-grained Style Specification and Attribute Consistency Reward [47.96949534259019]
  We propose a controllable dialogue generation model to steer response generation under multi-attribute constraints. We categorize the commonly used control attributes into global and local ones, which possess different granularities of effects on response generation. Our model can significantly outperform competitive baselines in terms of response quality, content diversity and controllability.
  arXiv  Detail & Related papers  (2021-09-14T14:29:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.