Tensor Component Analysis for Interpreting the Latent Space of GANs
- URL: http://arxiv.org/abs/2111.11736v1
- Date: Tue, 23 Nov 2021 09:14:39 GMT
- Title: Tensor Component Analysis for Interpreting the Latent Space of GANs
- Authors: James Oldfield, Markos Georgopoulos, Yannis Panagakis, Mihalis A.
Nicolaou, Ioannis Patras
- Abstract summary: This paper addresses the problem of finding interpretable directions in the latent space of pre-trained Generative Adversarial Networks (GANs)
Our scheme allows for both linear edits corresponding to the individual modes of the tensor, and non-linear ones that model the multiplicative interactions between them.
We show experimentally that we can utilise the former to better separate style- from geometry-based transformations, and the latter to generate an extended set of possible transformations.
- Score: 41.020230946351816
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper addresses the problem of finding interpretable directions in the
latent space of pre-trained Generative Adversarial Networks (GANs) to
facilitate controllable image synthesis. Such interpretable directions
correspond to transformations that can affect both the style and geometry of
the synthetic images. However, existing approaches that utilise linear
techniques to find these transformations often fail to provide an intuitive way
to separate these two sources of variation. To address this, we propose to a)
perform a multilinear decomposition of the tensor of intermediate
representations, and b) use a tensor-based regression to map directions found
using this decomposition to the latent space. Our scheme allows for both linear
edits corresponding to the individual modes of the tensor, and non-linear ones
that model the multiplicative interactions between them. We show experimentally
that we can utilise the former to better separate style- from geometry-based
transformations, and the latter to generate an extended set of possible
transformations in comparison to prior works. We demonstrate our approach's
efficacy both quantitatively and qualitatively compared to the current
state-of-the-art.
Related papers
- RLE: A Unified Perspective of Data Augmentation for Cross-Spectral Re-identification [59.5042031913258]
Non-linear modality discrepancy mainly comes from diverse linear transformations acting on the surface of different materials.
We propose a Random Linear Enhancement (RLE) strategy which includes Moderate Random Linear Enhancement (MRLE) and Radical Random Linear Enhancement (RRLE)
The experimental results not only demonstrate the superiority and effectiveness of RLE but also confirm its great potential as a general-purpose data augmentation for cross-spectral re-identification.
arXiv Detail & Related papers (2024-11-02T12:13:37Z) - Affine Invariance in Continuous-Domain Convolutional Neural Networks [6.019182604573028]
This research studies affine invariance on continuous-domain convolutional neural networks.
We introduce a new criterion to assess the similarity of two input signals under affine transformations.
Our research could eventually extend the scope of geometrical transformations that practical deep-learning pipelines can handle.
arXiv Detail & Related papers (2023-11-13T14:17:57Z) - A Theory of Topological Derivatives for Inverse Rendering of Geometry [87.49881303178061]
We introduce a theoretical framework for differentiable surface evolution that allows discrete topology changes through the use of topological derivatives.
We validate the proposed theory with optimization of closed curves in 2D and surfaces in 3D to lend insights into limitations of current methods.
arXiv Detail & Related papers (2023-08-19T00:55:55Z) - Accelerated MRI With Deep Linear Convolutional Transform Learning [7.927206441149002]
Recent studies show that deep learning based MRI reconstruction outperforms conventional methods in multiple applications.
In this work, we combine ideas from CS, TL and DL reconstructions to learn deep linear convolutional transforms.
Our results show that the proposed technique can reconstruct MR images to a level comparable to DL methods, while supporting uniform undersampling patterns.
arXiv Detail & Related papers (2022-04-17T04:47:32Z) - Topographic VAEs learn Equivariant Capsules [84.33745072274942]
We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables.
We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST.
We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
arXiv Detail & Related papers (2021-09-03T09:25:57Z) - Addressing the Topological Defects of Disentanglement via Distributed
Operators [10.29148285032989]
A popular approach to disentanglement consists in learning to map each of these factors to distinct subspaces of a model's latent representation.
Here, we show that for a broad family of transformations acting on images, this approach introduces topological defects.
Motivated by classical results from group representation theory, we study an alternative, more flexible approach to disentanglement.
arXiv Detail & Related papers (2021-02-10T18:34:55Z) - Joint Estimation of Image Representations and their Lie Invariants [57.3768308075675]
Images encode both the state of the world and its content.
The automatic extraction of this information is challenging because of the high-dimensionality and entangled encoding inherent to the image representation.
This article introduces two theoretical approaches aimed at the resolution of these challenges.
arXiv Detail & Related papers (2020-12-05T00:07:41Z) - Inverse Learning of Symmetries [71.62109774068064]
We learn the symmetry transformation with a model consisting of two latent subspaces.
Our approach is based on the deep information bottleneck in combination with a continuous mutual information regulariser.
Our model outperforms state-of-the-art methods on artificial and molecular datasets.
arXiv Detail & Related papers (2020-02-07T13:48:52Z) - Invertible Generative Modeling using Linear Rational Splines [11.510009152620666]
Normalizing flows attempt to model an arbitrary probability distribution through a set of invertible mappings.
The first flow designs used coupling layer mappings built upon affine transformations.
Intrepid piecewise functions as a replacement for affine transformations have attracted attention.
arXiv Detail & Related papers (2020-01-15T08:05:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.