GroupifyVAE: from Group-based Definition to VAE-based Unsupervised Representation Disentanglement

• URL: http://arxiv.org/abs/2102.10303v1
• Date: Sat, 20 Feb 2021 09:49:51 GMT
• Title: GroupifyVAE: from Group-based Definition to VAE-based Unsupervised Representation Disentanglement
• Authors: Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, Nanning Zheng, Pengju Ren
• Abstract summary: VAE-based unsupervised disentanglement can not be achieved without introducing other inductive bias. We address VAE-based unsupervised disentanglement by leveraging the constraints derived from the Group Theory based definition as the non-probabilistic inductive bias. We train 1800 models covering the most prominent VAE-based models on five datasets to verify the effectiveness of our method.
• Score: 91.9003001845855
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The key idea of the state-of-the-art VAE-based unsupervised representation disentanglement methods is to minimize the total correlation of the latent variable distributions. However, it has been proved that VAE-based unsupervised disentanglement can not be achieved without introducing other inductive bias. In this paper, we address VAE-based unsupervised disentanglement by leveraging the constraints derived from the Group Theory based definition as the non-probabilistic inductive bias. More specifically, inspired by the nth dihedral group (the permutation group for regular polygons), we propose a specific form of the definition and prove its two equivalent conditions: isomorphism and "the constancy of permutations". We further provide an implementation of isomorphism based on two Group constraints: the Abel constraint for the exchangeability and Order constraint for the cyclicity. We then convert them into a self-supervised training loss that can be incorporated into VAE-based models to bridge their gaps from the Group Theory based definition. We train 1800 models covering the most prominent VAE-based models on five datasets to verify the effectiveness of our method. Compared to the original models, the Groupidied VAEs consistently achieve better mean performance with smaller variances, and make meaningful dimensions controllable.

Related papers

• COD: Learning Conditional Invariant Representation for Domain Adaptation Regression [20.676363400841495]
  Domain Adaptation Regression is developed to generalize label knowledge from a source domain to an unlabeled target domain. Existing conditional distribution alignment theory and methods with discrete prior are no longer applicable. To minimize the discrepancy, a COD-based conditional invariant representation learning model is proposed.
  arXiv  Detail & Related papers  (2024-08-13T05:08:13Z)
• A Non-negative VAE:the Generalized Gamma Belief Network [49.970917207211556]
  The gamma belief network (GBN) has demonstrated its potential for uncovering multi-layer interpretable latent representations in text data. We introduce the generalized gamma belief network (Generalized GBN) in this paper, which extends the original linear generative model to a more expressive non-linear generative model. We also propose an upward-downward Weibull inference network to approximate the posterior distribution of the latent variables.
  arXiv  Detail & Related papers  (2024-08-06T18:18:37Z)
• Lie Group Decompositions for Equivariant Neural Networks [12.139222986297261]
  We show how convolution kernels can be parametrized to build models equivariant with respect to affine transformations. We evaluate the robustness and out-of-distribution generalisation capability of our model on the benchmark affine-invariant classification task.
  arXiv  Detail & Related papers  (2023-10-17T16:04:33Z)
• Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models [56.88106830869487]
  We introduce equi-tuning, a novel fine-tuning method that transforms (potentially non-equivariant) pretrained models into group equivariant models. We provide applications of equi-tuning on three different tasks: image classification, compositional generalization in language, and fairness in natural language generation.
  arXiv  Detail & Related papers  (2022-10-13T08:45:23Z)
• PAC Generalization via Invariant Representations [41.02828564338047]
  We consider the notion of $epsilon$-approximate invariance in a finite sample setting. Inspired by PAC learning, we obtain finite-sample out-of-distribution generalization guarantees. Our results show bounds that do not scale in ambient dimension when intervention sites are restricted to lie in a constant size subset of in-degree bounded nodes.
  arXiv  Detail & Related papers  (2022-05-30T15:50:14Z)
• Commutative Lie Group VAE for Disentanglement Learning [96.32813624341833]
  We view disentanglement learning as discovering an underlying structure that equivariantly reflects the factorized variations shown in data. A simple model named Commutative Lie Group VAE is introduced to realize the group-based disentanglement learning. Experiments show that our model can effectively learn disentangled representations without supervision, and can achieve state-of-the-art performance without extra constraints.
  arXiv  Detail & Related papers  (2021-06-07T07:03:14Z)
• LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
  Group equivariant neural networks are used as building blocks of group invariant neural networks. We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
  arXiv  Detail & Related papers  (2020-12-20T11:02:49Z)
• Posterior Differential Regularization with f-divergence for Improving Model Robustness [95.05725916287376]
  We focus on methods that regularize the model posterior difference between clean and noisy inputs. We generalize the posterior differential regularization to the family of $f$-divergences. Our experiments show that regularizing the posterior differential with $f$-divergence can result in well-improved model robustness.
  arXiv  Detail & Related papers  (2020-10-23T19:58:01Z)
• An Identifiable Double VAE For Disentangled Representations [24.963285614606665]
  We propose a novel VAE-based generative model with theoretical guarantees on identifiability. We obtain our conditional prior over the latents by learning an optimal representation. Experimental results indicate superior performance with respect to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-10-19T09:59:31Z)
• Variational Intrinsic Control Revisited [7.6146285961466]
  In the original work by Gregor et al., two VIC algorithms were proposed: one that represents the options explicitly, and the other that does it implicitly. We show that the intrinsic reward used in the latter is subject to bias in environments, causing convergence to suboptimal solutions. We propose two methods to correct this behavior and achieve the maximal empowerment.
  arXiv  Detail & Related papers  (2020-10-07T09:00:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.