GroupifyVAE: from Group-based Definition to VAE-based Unsupervised
Representation Disentanglement
- URL: http://arxiv.org/abs/2102.10303v1
- Date: Sat, 20 Feb 2021 09:49:51 GMT
- Title: GroupifyVAE: from Group-based Definition to VAE-based Unsupervised
Representation Disentanglement
- Authors: Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, Nanning Zheng, Pengju
Ren
- Abstract summary: VAE-based unsupervised disentanglement can not be achieved without introducing other inductive bias.
We address VAE-based unsupervised disentanglement by leveraging the constraints derived from the Group Theory based definition as the non-probabilistic inductive bias.
We train 1800 models covering the most prominent VAE-based models on five datasets to verify the effectiveness of our method.
- Score: 91.9003001845855
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The key idea of the state-of-the-art VAE-based unsupervised representation
disentanglement methods is to minimize the total correlation of the latent
variable distributions. However, it has been proved that VAE-based unsupervised
disentanglement can not be achieved without introducing other inductive bias.
In this paper, we address VAE-based unsupervised disentanglement by leveraging
the constraints derived from the Group Theory based definition as the
non-probabilistic inductive bias. More specifically, inspired by the nth
dihedral group (the permutation group for regular polygons), we propose a
specific form of the definition and prove its two equivalent conditions:
isomorphism and "the constancy of permutations". We further provide an
implementation of isomorphism based on two Group constraints: the Abel
constraint for the exchangeability and Order constraint for the cyclicity. We
then convert them into a self-supervised training loss that can be incorporated
into VAE-based models to bridge their gaps from the Group Theory based
definition. We train 1800 models covering the most prominent VAE-based models
on five datasets to verify the effectiveness of our method. Compared to the
original models, the Groupidied VAEs consistently achieve better mean
performance with smaller variances, and make meaningful dimensions
controllable.
Related papers
- Lie Group Decompositions for Equivariant Neural Networks [12.139222986297261]
We show how convolution kernels can be parametrized to build models equivariant with respect to affine transformations.
We evaluate the robustness and out-of-distribution generalisation capability of our model on the benchmark affine-invariant classification task.
arXiv Detail & Related papers (2023-10-17T16:04:33Z) - Representation Disentaglement via Regularization by Causal
Identification [3.9160947065896803]
We propose the use of a causal collider structured model to describe the underlying data generative process assumptions in disentangled representation learning.
For this, we propose regularization by identification (ReI), a modular regularization engine designed to align the behavior of large scale generative models with the disentanglement constraints imposed by causal identification.
arXiv Detail & Related papers (2023-02-28T23:18:54Z) - Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models [56.88106830869487]
We introduce equi-tuning, a novel fine-tuning method that transforms (potentially non-equivariant) pretrained models into group equivariant models.
We provide applications of equi-tuning on three different tasks: image classification, compositional generalization in language, and fairness in natural language generation.
arXiv Detail & Related papers (2022-10-13T08:45:23Z) - PAC Generalization via Invariant Representations [41.02828564338047]
We consider the notion of $epsilon$-approximate invariance in a finite sample setting.
Inspired by PAC learning, we obtain finite-sample out-of-distribution generalization guarantees.
Our results show bounds that do not scale in ambient dimension when intervention sites are restricted to lie in a constant size subset of in-degree bounded nodes.
arXiv Detail & Related papers (2022-05-30T15:50:14Z) - Regularizing Variational Autoencoder with Diversity and Uncertainty
Awareness [61.827054365139645]
Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference.
We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
arXiv Detail & Related papers (2021-10-24T07:58:13Z) - Commutative Lie Group VAE for Disentanglement Learning [96.32813624341833]
We view disentanglement learning as discovering an underlying structure that equivariantly reflects the factorized variations shown in data.
A simple model named Commutative Lie Group VAE is introduced to realize the group-based disentanglement learning.
Experiments show that our model can effectively learn disentangled representations without supervision, and can achieve state-of-the-art performance without extra constraints.
arXiv Detail & Related papers (2021-06-07T07:03:14Z) - LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
Group equivariant neural networks are used as building blocks of group invariant neural networks.
We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models.
We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
arXiv Detail & Related papers (2020-12-20T11:02:49Z) - Posterior Differential Regularization with f-divergence for Improving
Model Robustness [95.05725916287376]
We focus on methods that regularize the model posterior difference between clean and noisy inputs.
We generalize the posterior differential regularization to the family of $f$-divergences.
Our experiments show that regularizing the posterior differential with $f$-divergence can result in well-improved model robustness.
arXiv Detail & Related papers (2020-10-23T19:58:01Z) - An Identifiable Double VAE For Disentangled Representations [24.963285614606665]
We propose a novel VAE-based generative model with theoretical guarantees on identifiability.
We obtain our conditional prior over the latents by learning an optimal representation.
Experimental results indicate superior performance with respect to state-of-the-art approaches.
arXiv Detail & Related papers (2020-10-19T09:59:31Z) - Variational Intrinsic Control Revisited [7.6146285961466]
In the original work by Gregor et al., two VIC algorithms were proposed: one that represents the options explicitly, and the other that does it implicitly.
We show that the intrinsic reward used in the latter is subject to bias in environments, causing convergence to suboptimal solutions.
We propose two methods to correct this behavior and achieve the maximal empowerment.
arXiv Detail & Related papers (2020-10-07T09:00:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.