Tensor Component Analysis for Interpreting the Latent Space of GANs

• URL: http://arxiv.org/abs/2111.11736v1
• Date: Tue, 23 Nov 2021 09:14:39 GMT
• Title: Tensor Component Analysis for Interpreting the Latent Space of GANs
• Authors: James Oldfield, Markos Georgopoulos, Yannis Panagakis, Mihalis A. Nicolaou, Ioannis Patras
• Abstract summary: This paper addresses the problem of finding interpretable directions in the latent space of pre-trained Generative Adversarial Networks (GANs) Our scheme allows for both linear edits corresponding to the individual modes of the tensor, and non-linear ones that model the multiplicative interactions between them. We show experimentally that we can utilise the former to better separate style- from geometry-based transformations, and the latter to generate an extended set of possible transformations.
• Score: 41.020230946351816
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper addresses the problem of finding interpretable directions in the latent space of pre-trained Generative Adversarial Networks (GANs) to facilitate controllable image synthesis. Such interpretable directions correspond to transformations that can affect both the style and geometry of the synthetic images. However, existing approaches that utilise linear techniques to find these transformations often fail to provide an intuitive way to separate these two sources of variation. To address this, we propose to a) perform a multilinear decomposition of the tensor of intermediate representations, and b) use a tensor-based regression to map directions found using this decomposition to the latent space. Our scheme allows for both linear edits corresponding to the individual modes of the tensor, and non-linear ones that model the multiplicative interactions between them. We show experimentally that we can utilise the former to better separate style- from geometry-based transformations, and the latter to generate an extended set of possible transformations in comparison to prior works. We demonstrate our approach's efficacy both quantitatively and qualitatively compared to the current state-of-the-art.

Related papers

• Affine Invariance in Continuous-Domain Convolutional Neural Networks [6.019182604573028]
  This research studies affine invariance on continuous-domain convolutional neural networks. We introduce a new criterion to assess the similarity of two input signals under affine transformations. Our research could eventually extend the scope of geometrical transformations that practical deep-learning pipelines can handle.
  arXiv  Detail & Related papers  (2023-11-13T14:17:57Z)
• A Theory of Topological Derivatives for Inverse Rendering of Geometry [87.49881303178061]
  We introduce a theoretical framework for differentiable surface evolution that allows discrete topology changes through the use of topological derivatives. We validate the proposed theory with optimization of closed curves in 2D and surfaces in 3D to lend insights into limitations of current methods.
  arXiv  Detail & Related papers  (2023-08-19T00:55:55Z)
• Accelerated MRI With Deep Linear Convolutional Transform Learning [7.927206441149002]
  Recent studies show that deep learning based MRI reconstruction outperforms conventional methods in multiple applications. In this work, we combine ideas from CS, TL and DL reconstructions to learn deep linear convolutional transforms. Our results show that the proposed technique can reconstruct MR images to a level comparable to DL methods, while supporting uniform undersampling patterns.
  arXiv  Detail & Related papers  (2022-04-17T04:47:32Z)
• Topographic VAEs learn Equivariant Capsules [84.33745072274942]
  We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
  arXiv  Detail & Related papers  (2021-09-03T09:25:57Z)
• Self-Supervised Graph Representation Learning via Topology Transformations [61.870882736758624]
  We present the Topology Transformation Equivariant Representation learning, a general paradigm of self-supervised learning for node representations of graph data. In experiments, we apply the proposed model to the downstream node and graph classification tasks, and results show that the proposed method outperforms the state-of-the-art unsupervised approaches.
  arXiv  Detail & Related papers  (2021-05-25T06:11:03Z)
• Addressing the Topological Defects of Disentanglement via Distributed Operators [10.29148285032989]
  A popular approach to disentanglement consists in learning to map each of these factors to distinct subspaces of a model's latent representation. Here, we show that for a broad family of transformations acting on images, this approach introduces topological defects. Motivated by classical results from group representation theory, we study an alternative, more flexible approach to disentanglement.
  arXiv  Detail & Related papers  (2021-02-10T18:34:55Z)
• Joint Estimation of Image Representations and their Lie Invariants [57.3768308075675]
  Images encode both the state of the world and its content. The automatic extraction of this information is challenging because of the high-dimensionality and entangled encoding inherent to the image representation. This article introduces two theoretical approaches aimed at the resolution of these challenges.
  arXiv  Detail & Related papers  (2020-12-05T00:07:41Z)
• Inverse Learning of Symmetries [71.62109774068064]
  We learn the symmetry transformation with a model consisting of two latent subspaces. Our approach is based on the deep information bottleneck in combination with a continuous mutual information regulariser. Our model outperforms state-of-the-art methods on artificial and molecular datasets.
  arXiv  Detail & Related papers  (2020-02-07T13:48:52Z)
• Invertible Generative Modeling using Linear Rational Splines [11.510009152620666]
  Normalizing flows attempt to model an arbitrary probability distribution through a set of invertible mappings. The first flow designs used coupling layer mappings built upon affine transformations. Intrepid piecewise functions as a replacement for affine transformations have attracted attention.
  arXiv  Detail & Related papers  (2020-01-15T08:05:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.