GAN "Steerability" without optimization

• URL: http://arxiv.org/abs/2012.05328v2
• Date: Sun, 24 Jan 2021 16:50:39 GMT
• Title: GAN "Steerability" without optimization
• Authors: Nurit Spingarn-Eliezer, Ron Banner and Tomer Michaeli
• Abstract summary: "steering" directions correspond to semantically meaningful image transformations. We show that "steering" trajectories can be computed in closed form directly from the generator's weights.
• Score: 32.63317794951011
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent research has shown remarkable success in revealing "steering" directions in the latent spaces of pre-trained GANs. These directions correspond to semantically meaningful image transformations e.g., shift, zoom, color manipulations), and have similar interpretable effects across all categories that the GAN can generate. Some methods focus on user-specified transformations, while others discover transformations in an unsupervised manner. However, all existing techniques rely on an optimization procedure to expose those directions, and offer no control over the degree of allowed interaction between different transformations. In this paper, we show that "steering" trajectories can be computed in closed form directly from the generator's weights without any form of training or optimization. This applies to user-prescribed geometric transformations, as well as to unsupervised discovery of more complex effects. Our approach allows determining both linear and nonlinear trajectories, and has many advantages over previous methods. In particular, we can control whether one transformation is allowed to come on the expense of another (e.g. zoom-in with or without allowing translation to keep the object centered). Moreover, we can determine the natural end-point of the trajectory, which corresponds to the largest extent to which a transformation can be applied without incurring degradation. Finally, we show how transferring attributes between images can be achieved without optimization, even across different categories.

Related papers

• Surrogate-Based Differentiable Pipeline for Shape Optimization [64.24199762940444]
  We propose replacing non-differentiable pipeline components with surrogate models which are inherently differentiable.<n>We demonstrate an end-to-end differentiable pipeline where a 3D U-Net full-field surrogate replaces both meshing and simulation steps by training it on the mapping between the signed distance field (SDF) of the shape and the fields of interest.
  arXiv  Detail & Related papers  (2025-11-13T19:30:50Z)
• Self-supervised Transformation Learning for Equivariant Representations [26.207358743969277]
  Unsupervised representation learning has significantly advanced various machine learning tasks. We propose Self-supervised Transformation Learning (STL), replacing transformation labels with transformation representations derived from image pairs. We demonstrate the approach's effectiveness across diverse classification and detection tasks, outperforming existing methods in 7 out of 11 benchmarks.
  arXiv  Detail & Related papers  (2025-01-15T10:54:21Z)
• Adaptive Nonlinear Latent Transformation for Conditional Face Editing [40.32385363670918]
  We propose a novel adaptive nonlinear latent transformation for disentangled and conditional face editing, termed AdaTrans. AdaTrans divides the manipulation process into several finer steps; i.e., the direction and size at each step are conditioned on both the facial attributes and the latent codes. AdaTrans enables a controllable face editing with the advantages of disentanglement, flexibility with non-binary attributes, and high fidelity.
  arXiv  Detail & Related papers  (2023-07-15T12:36:50Z)
• ParGAN: Learning Real Parametrizable Transformations [50.51405390150066]
  We propose ParGAN, a generalization of the cycle-consistent GAN framework to learn image transformations. The proposed generator takes as input both an image and a parametrization of the transformation. We show how, with disjoint image domains with no annotated parametrization, our framework can create smooths as well as learn multiple transformations simultaneously.
  arXiv  Detail & Related papers  (2022-11-09T16:16:06Z)
• Overparameterization Improves StyleGAN Inversion [66.8300251627992]
  Existing inversion approaches obtain promising yet imperfect results. We show that this allows us to obtain near-perfect image reconstruction without the need for encoders. Our approach also retains editability, which we demonstrate by realistically interpolating between images.
  arXiv  Detail & Related papers  (2022-05-12T18:42:43Z)
• Revisiting Transformation Invariant Geometric Deep Learning: Are Initial Representations All You Need? [80.86819657126041]
  We show that transformation-invariant and distance-preserving initial representations are sufficient to achieve transformation invariance. Specifically, we realize transformation-invariant and distance-preserving initial point representations by modifying multi-dimensional scaling. We prove that TinvNN can strictly guarantee transformation invariance, being general and flexible enough to be combined with the existing neural networks.
  arXiv  Detail & Related papers  (2021-12-23T03:52:33Z)
• Tensor Component Analysis for Interpreting the Latent Space of GANs [41.020230946351816]
  This paper addresses the problem of finding interpretable directions in the latent space of pre-trained Generative Adversarial Networks (GANs) Our scheme allows for both linear edits corresponding to the individual modes of the tensor, and non-linear ones that model the multiplicative interactions between them. We show experimentally that we can utilise the former to better separate style- from geometry-based transformations, and the latter to generate an extended set of possible transformations.
  arXiv  Detail & Related papers  (2021-11-23T09:14:39Z)
• Unsupervised Discovery of Disentangled Manifolds in GANs [74.24771216154105]
  Interpretable generation process is beneficial to various image editing applications. We propose a framework to discover interpretable directions in the latent space given arbitrary pre-trained generative adversarial networks.
  arXiv  Detail & Related papers  (2020-11-24T02:18:08Z)
• Data Augmentation via Structured Adversarial Perturbations [25.31035665982414]
  We propose a method to generate adversarial examples that maintain some desired natural structure. We demonstrate this approach through two types of image transformations: photometric and geometric.
  arXiv  Detail & Related papers  (2020-11-05T18:07:55Z)
• Channel-Directed Gradients for Optimization of Convolutional Neural Networks [50.34913837546743]
  We introduce optimization methods for convolutional neural networks that can be used to improve existing gradient-based optimization in terms of generalization error. We show that defining the gradients along the output channel direction leads to a performance boost, while other directions can be detrimental.
  arXiv  Detail & Related papers  (2020-08-25T00:44:09Z)
• Deriving Differential Target Propagation from Iterating Approximate Inverses [91.3755431537592]
  We show that a particular form of target propagation, relying on learned inverses of each layer, which is differential, gives rise to an update rule which corresponds to an approximate Gauss-Newton gradient-based optimization. We consider several iterative calculations based on local auto-encoders at each layer in order to achieve more precise inversions for more accurate target propagation.
  arXiv  Detail & Related papers  (2020-07-29T22:34:45Z)
• A Novel Graphic Bending Transformation on Benchmark [6.6326947833070395]
  We investigate a novel graphic conformal mapping transformation on benchmark problems to deform the function shape. Experiments indicate the same spends more search budget and encounter more failures on the conformal bent functions than the rotated version.
  arXiv  Detail & Related papers  (2020-04-21T14:31:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.