Exploring Attribute Variations in Style-based GANs using Diffusion Models

• URL: http://arxiv.org/abs/2311.16052v1
• Date: Mon, 27 Nov 2023 18:14:03 GMT
• Title: Exploring Attribute Variations in Style-based GANs using Diffusion Models
• Authors: Rishubh Parihar, Prasanna Balaji, Raghav Magazine, Sarthak Vora, Tejan Karmali, Varun Jampani, R. Venkatesh Babu
• Abstract summary: We formulate the task of textitdiverse attribute editing by modeling the multidimensional nature of attribute edits. We capitalize on disentangled latent spaces of pretrained GANs and train a Denoising Diffusion Probabilistic Model (DDPM) to learn the latent distribution for diverse edits.
• Score: 48.98081892627042
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing attribute editing methods treat semantic attributes as binary, resulting in a single edit per attribute. However, attributes such as eyeglasses, smiles, or hairstyles exhibit a vast range of diversity. In this work, we formulate the task of \textit{diverse attribute editing} by modeling the multidimensional nature of attribute edits. This enables users to generate multiple plausible edits per attribute. We capitalize on disentangled latent spaces of pretrained GANs and train a Denoising Diffusion Probabilistic Model (DDPM) to learn the latent distribution for diverse edits. Specifically, we train DDPM over a dataset of edit latent directions obtained by embedding image pairs with a single attribute change. This leads to latent subspaces that enable diverse attribute editing. Applying diffusion in the highly compressed latent space allows us to model rich distributions of edits within limited computational resources. Through extensive qualitative and quantitative experiments conducted across a range of datasets, we demonstrate the effectiveness of our approach for diverse attribute editing. We also showcase the results of our method applied for 3D editing of various face attributes.

Related papers

• DreamMix: Decoupling Object Attributes for Enhanced Editability in Customized Image Inpainting [63.01425442236011]
  We present DreamMix, a diffusion-based generative model adept at inserting target objects into scenes at user-specified locations. We propose an Attribute Decoupling Mechanism (ADM) and a Textual Attribute Substitution (TAS) module to improve the diversity and discriminative capability of the text-based attribute guidance.
  arXiv  Detail & Related papers  (2024-11-26T08:44:47Z)
• AttriHuman-3D: Editable 3D Human Avatar Generation with Attribute Decomposition and Indexing [79.38471599977011]
  We propose AttriHuman-3D, an editable 3D human generation model. It generates all attributes in an overall attribute space with six feature planes, which are decomposed and manipulated with different attribute indexes. Our model provides a strong disentanglement between different attributes, allows fine-grained image editing and generates high-quality 3D human avatars.
  arXiv  Detail & Related papers  (2023-12-03T03:20:10Z)
• Multi-Directional Subspace Editing in Style-Space [6.282068591820945]
  This paper describes a new technique for finding disentangled semantic directions in the latent space of StyleGAN. Our model is capable of editing a single attribute in multiple directions, resulting in a range of possible generated images.
  arXiv  Detail & Related papers  (2022-11-21T19:47:35Z)
• Leveraging Off-the-shelf Diffusion Model for Multi-attribute Fashion Image Manipulation [27.587905673112473]
  Fashion attribute editing is a task that aims to convert the semantic attributes of a given fashion image while preserving the irrelevant regions. Previous works typically employ conditional GANs where the generator explicitly learns the target attributes and directly execute the conversion. We explore the classifier-guided diffusion that leverages the off-the-shelf diffusion model pretrained on general visual semantics such as Imagenet.
  arXiv  Detail & Related papers  (2022-10-12T02:21:18Z)
• ManiCLIP: Multi-Attribute Face Manipulation from Text [104.30600573306991]
  We present a novel multi-attribute face manipulation method based on textual descriptions. Our method generates natural manipulated faces with minimal text-irrelevant attribute editing.
  arXiv  Detail & Related papers  (2022-10-02T07:22:55Z)
• Everything is There in Latent Space: Attribute Editing and Attribute Style Manipulation by StyleGAN Latent Space Exploration [39.18239951479647]
  We present Few-shot Latent-based Attribute Manipulation and Editing (FLAME) FLAME is a framework to perform highly controlled image editing by latent space manipulation. We generate diverse attribute styles in disentangled manner.
  arXiv  Detail & Related papers  (2022-07-20T12:40:32Z)
• Each Attribute Matters: Contrastive Attention for Sentence-based Image Editing [13.321782757637303]
  Sentence-based Image Editing (SIE) aims to deploy natural language to edit an image. Existing methods can hardly produce accurate editing when the query sentence is with multiple editable attributes. This paper proposes a novel model called Contrastive Attention Generative Adversarial Network (CA-GAN)
  arXiv  Detail & Related papers  (2021-10-21T14:06:20Z)
• Disentangled Face Attribute Editing via Instance-Aware Latent Space Search [30.17338705964925]
  A rich set of semantic directions exist in the latent space of Generative Adversarial Networks (GANs) Existing methods may suffer poor attribute variation disentanglement, leading to unwanted change of other attributes when altering the desired one. We propose a novel framework (IALS) that performs Instance-Aware Latent-Space Search to find semantic directions for disentangled attribute editing.
  arXiv  Detail & Related papers  (2021-05-26T16:19:08Z)
• SMILE: Semantically-guided Multi-attribute Image and Layout Editing [154.69452301122175]
  Attribute image manipulation has been a very active topic since the introduction of Generative Adversarial Networks (GANs) We present a multimodal representation that handles all attributes, be it guided by random noise or images, while only using the underlying domain information of the target domain. Our method is capable of adding, removing or changing either fine-grained or coarse attributes by using an image as a reference or by exploring the style distribution space.
  arXiv  Detail & Related papers  (2020-10-05T20:15:21Z)
• Attribute-based Regularization of Latent Spaces for Variational Auto-Encoders [79.68916470119743]
  We present a novel method to structure the latent space of a Variational Auto-Encoder (VAE) to encode different continuous-valued attributes explicitly. This is accomplished by using an attribute regularization loss which enforces a monotonic relationship between the attribute values and the latent code of the dimension along which the attribute is to be encoded.
  arXiv  Detail & Related papers  (2020-04-11T20:53:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.