Rayleigh EigenDirections (REDs): GAN latent space traversals for multidimensional features

• URL: http://arxiv.org/abs/2201.10423v1
• Date: Tue, 25 Jan 2022 16:11:33 GMT
• Title: Rayleigh EigenDirections (REDs): GAN latent space traversals for multidimensional features
• Authors: Guha Balakrishnan, Raghudeep Gadde, Aleix Martinez, Pietro Perona
• Abstract summary: We present a method for finding paths in a deep generative model's latent space. We can manipulate multidimensional features of an image such as facial identity and pixels within a region. Our work suggests that a wealth of opportunities lies in the local analysis of the geometry and semantics of latent spaces.
• Score: 20.11085769303415
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a method for finding paths in a deep generative model's latent space that can maximally vary one set of image features while holding others constant. Crucially, unlike past traversal approaches, ours can manipulate multidimensional features of an image such as facial identity and pixels within a specified region. Our method is principled and conceptually simple: optimal traversal directions are chosen by maximizing differential changes to one feature set such that changes to another set are negligible. We show that this problem is nearly equivalent to one of Rayleigh quotient maximization, and provide a closed-form solution to it based on solving a generalized eigenvalue equation. We use repeated computations of the corresponding optimal directions, which we call Rayleigh EigenDirections (REDs), to generate appropriately curved paths in latent space. We empirically evaluate our method using StyleGAN2 on two image domains: faces and living rooms. We show that our method is capable of controlling various multidimensional features out of the scope of previous latent space traversal methods: face identity, spatial frequency bands, pixels within a region, and the appearance and position of an object. Our work suggests that a wealth of opportunities lies in the local analysis of the geometry and semantics of latent spaces.

Related papers

• Probabilistic Directed Distance Fields for Ray-Based Shape Representations [8.134429779950658]
  Directed Distance Fields (DDFs) are a novel neural shape representation that builds upon classical distance fields. We show how to model inherent discontinuities in the underlying field. We then apply DDFs to several applications, including single-shape fitting, generative modelling, and single-image 3D reconstruction.
  arXiv  Detail & Related papers  (2024-04-13T21:02:49Z)
• $PC^2$: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction [97.06927852165464]
  Reconstructing the 3D shape of an object from a single RGB image is a long-standing and highly challenging problem in computer vision. We propose a novel method for single-image 3D reconstruction which generates a sparse point cloud via a conditional denoising diffusion process.
  arXiv  Detail & Related papers  (2023-02-21T13:37:07Z)
• WarpedGANSpace: Finding non-linear RBF paths in GAN latent space [44.7091944340362]
  This work addresses the problem of discovering, in an unsupervised manner, interpretable paths in the latent space of pretrained GANs. We learn non-linear warpings on the latent space, each one parametrized by a set of RBF-based latent space warping functions. We show that linear paths can be derived as a special case of our method, and show experimentally that non-linear paths in the latent space lead to steeper, more disentangled and interpretable changes in the image space.
  arXiv  Detail & Related papers  (2021-09-27T21:29:35Z)
• Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization [59.19214040221055]
  We propose a novel spatial-separated curve rendering network (S$2$CRNet) for efficient and high-resolution image harmonization. The proposed method reduces more than 90% parameters compared with previous methods. Our method can work smoothly on higher resolution images in real-time which is more than 10$times$ faster than the existing methods.
  arXiv  Detail & Related papers  (2021-09-13T07:20:16Z)
• Semantic and Geometric Unfolding of StyleGAN Latent Space [2.7910505923792646]
  Generative adversarial networks (GANs) have proven to be surprisingly efficient for image editing. In this paper, we identify two geometric limitations of such latent space. We propose a new method to learn a proxy latent representation using normalizing flows to remedy these limitations.
  arXiv  Detail & Related papers  (2021-07-09T15:12:55Z)
• DeepMesh: Differentiable Iso-Surface Extraction [53.77622255726208]
  We introduce a differentiable way to produce explicit surface mesh representations from Deep Implicit Fields. Our key insight is that by reasoning on how implicit field perturbations impact local surface geometry, one can ultimately differentiate the 3D location of surface samples. We exploit this to define DeepMesh -- end-to-end differentiable mesh representation that can vary its topology.
  arXiv  Detail & Related papers  (2021-06-20T20:12:41Z)
• Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs [7.443321740418409]
  We propose a method to find local-geometry-aware traversal directions on the intermediate latent space of Generative Adversarial Networks (GANs) Motivated by the intrinsic sparsity of the latent space, the basis is discovered by solving the low-rank approximation problem of the differential of the partial network.
  arXiv  Detail & Related papers  (2021-06-13T10:29:42Z)
• Low-Rank Subspaces in GANs [101.48350547067628]
  This work introduces low-rank subspaces that enable more precise control of GAN generation. LowRankGAN is able to find the low-dimensional representation of attribute manifold. Experiments on state-of-the-art GAN models (including StyleGAN2 and BigGAN) trained on various datasets demonstrate the effectiveness of our LowRankGAN.
  arXiv  Detail & Related papers  (2021-06-08T16:16:32Z)
• ShaRF: Shape-conditioned Radiance Fields from a Single View [54.39347002226309]
  We present a method for estimating neural scenes representations of objects given only a single image. The core of our method is the estimation of a geometric scaffold for the object. We demonstrate in several experiments the effectiveness of our approach in both synthetic and real images.
  arXiv  Detail & Related papers  (2021-02-17T16:40:28Z)
• The Geometry of Deep Generative Image Models and its Applications [0.0]
  Generative adversarial networks (GANs) have emerged as a powerful unsupervised method to model the statistical patterns of real-world data sets. These networks are trained to map random inputs in their latent space to new samples representative of the learned data. The structure of the latent space is hard to intuit due to its high dimensionality and the non-linearity of the generator.
  arXiv  Detail & Related papers  (2021-01-15T07:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.