SC2GAN: Rethinking Entanglement by Self-correcting Correlated GAN Space

• URL: http://arxiv.org/abs/2310.06667v1
• Date: Tue, 10 Oct 2023 14:42:32 GMT
• Title: SC2GAN: Rethinking Entanglement by Self-correcting Correlated GAN Space
• Authors: Zikun Chen, Han Zhao, Parham Aarabi, Ruowei Jiang
• Abstract summary: Gene Networks that achieve following editing directions for one attribute could result in entangled changes with other attributes. We propose a novel framework SC$2$GAN disentanglement by re-projecting low-density latent code samples in the original latent space.
• Score: 16.040942072859075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative Adversarial Networks (GANs) can synthesize realistic images, with the learned latent space shown to encode rich semantic information with various interpretable directions. However, due to the unstructured nature of the learned latent space, it inherits the bias from the training data where specific groups of visual attributes that are not causally related tend to appear together, a phenomenon also known as spurious correlations, e.g., age and eyeglasses or women and lipsticks. Consequently, the learned distribution often lacks the proper modelling of the missing examples. The interpolation following editing directions for one attribute could result in entangled changes with other attributes. To address this problem, previous works typically adjust the learned directions to minimize the changes in other attributes, yet they still fail on strongly correlated features. In this work, we study the entanglement issue in both the training data and the learned latent space for the StyleGAN2-FFHQ model. We propose a novel framework SC$^2$GAN that achieves disentanglement by re-projecting low-density latent code samples in the original latent space and correcting the editing directions based on both the high-density and low-density regions. By leveraging the original meaningful directions and semantic region-specific layers, our framework interpolates the original latent codes to generate images with attribute combination that appears infrequently, then inverts these samples back to the original latent space. We apply our framework to pre-existing methods that learn meaningful latent directions and showcase its strong capability to disentangle the attributes with small amounts of low-density region samples added.

Related papers

• Exploring Diffusion Time-steps for Unsupervised Representation Learning [72.43246871893936]
  We build a theoretical framework that connects the diffusion time-steps and the hidden attributes. On CelebA, FFHQ, and Bedroom datasets, the learned feature significantly improves classification.
  arXiv  Detail & Related papers  (2024-01-21T08:35:25Z)
• Spatial Steerability of GANs via Self-Supervision from Discriminator [123.27117057804732]
  We propose a self-supervised approach to improve the spatial steerability of GANs without searching for steerable directions in the latent space. Specifically, we design randomly sampled Gaussian heatmaps to be encoded into the intermediate layers of generative models as spatial inductive bias. During inference, users can interact with the spatial heatmaps in an intuitive manner, enabling them to edit the output image by adjusting the scene layout, moving, or removing objects.
  arXiv  Detail & Related papers  (2023-01-20T07:36:29Z)
• Discovering Class-Specific GAN Controls for Semantic Image Synthesis [73.91655061467988]
  We propose a novel method for finding spatially disentangled class-specific directions in the latent space of pretrained SIS models. We show that the latent directions found by our method can effectively control the local appearance of semantic classes.
  arXiv  Detail & Related papers  (2022-12-02T21:39:26Z)
• Exploring Gradient-based Multi-directional Controls in GANs [19.950198707910587]
  We propose a novel approach that discovers nonlinear controls, which enables multi-directional manipulation as well as effective disentanglement. Our approach is able to gain fine-grained controls over a diverse set of bi-directional and multi-directional attributes, and we showcase its ability to achieve disentanglement significantly better than state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-09-01T19:10:26Z)
• GSMFlow: Generation Shifts Mitigating Flow for Generalized Zero-Shot Learning [55.79997930181418]
  Generalized Zero-Shot Learning aims to recognize images from both the seen and unseen classes by transferring semantic knowledge from seen to unseen classes. It is a promising solution to take the advantage of generative models to hallucinate realistic unseen samples based on the knowledge learned from the seen classes. We propose a novel flow-based generative framework that consists of multiple conditional affine coupling layers for learning unseen data generation.
  arXiv  Detail & Related papers  (2022-07-05T04:04:37Z)
• Latent Transformations via NeuralODEs for GAN-based Image Editing [25.272389610447856]
  We show that nonlinear latent code manipulations realized as flows of a trainable Neural ODE are beneficial for many practical non-face image domains. In particular, we investigate a large number of datasets with known attributes and demonstrate that certain attribute manipulations are challenging to obtain with linear shifts only.
  arXiv  Detail & Related papers  (2021-11-29T18:59:54Z)
• LARGE: Latent-Based Regression through GAN Semantics [42.50535188836529]
  We propose a novel method for solving regression tasks using few-shot or weak supervision. We show that our method can be applied across a wide range of domains, leverage multiple latent direction discovery frameworks, and achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2021-07-22T17:55:35Z)
• Mitigating Generation Shifts for Generalized Zero-Shot Learning [52.98182124310114]
  Generalized Zero-Shot Learning (GZSL) is the task of leveraging semantic information (e.g., attributes) to recognize the seen and unseen samples, where unseen classes are not observable during training. We propose a novel Generation Shifts Mitigating Flow framework for learning unseen data synthesis efficiently and effectively. Experimental results demonstrate that GSMFlow achieves state-of-the-art recognition performance in both conventional and generalized zero-shot settings.
  arXiv  Detail & Related papers  (2021-07-07T11:43:59Z)
• Low-Rank Subspaces in GANs [101.48350547067628]
  This work introduces low-rank subspaces that enable more precise control of GAN generation. LowRankGAN is able to find the low-dimensional representation of attribute manifold. Experiments on state-of-the-art GAN models (including StyleGAN2 and BigGAN) trained on various datasets demonstrate the effectiveness of our LowRankGAN.
  arXiv  Detail & Related papers  (2021-06-08T16:16:32Z)
• Disentangled Face Attribute Editing via Instance-Aware Latent Space Search [30.17338705964925]
  A rich set of semantic directions exist in the latent space of Generative Adversarial Networks (GANs) Existing methods may suffer poor attribute variation disentanglement, leading to unwanted change of other attributes when altering the desired one. We propose a novel framework (IALS) that performs Instance-Aware Latent-Space Search to find semantic directions for disentangled attribute editing.
  arXiv  Detail & Related papers  (2021-05-26T16:19:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.