Variantional autoencoder with decremental information bottleneck for disentanglement

• URL: http://arxiv.org/abs/2303.12959v2
• Date: Wed, 4 Oct 2023 13:20:06 GMT
• Title: Variantional autoencoder with decremental information bottleneck for disentanglement
• Authors: Jiantao Wu, Shentong Mo, Xiang Yang, Muhammad Awais, Sara Atito, Xingshen Zhang, Lin Wang, Xiang Yang
• Abstract summary: We present a novel framework for disentangled representation learning, DeVAE, which utilizes hierarchical latent spaces with decreasing information bottlenecks. The key innovation of our approach lies in connecting the hierarchical latent spaces through disentanglement-invariant transformations. We demonstrate the effectiveness of DeVAE in achieving a balance between disentanglement and reconstruction through a series of experiments and ablation studies on dSprites and Shapes3D datasets.
• Score: 16.93743613675349
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One major challenge of disentanglement learning with variational autoencoders is the trade-off between disentanglement and reconstruction fidelity. Previous studies, which increase the information bottleneck during training, tend to lose the constraint of disentanglement, leading to the information diffusion problem. In this paper, we present a novel framework for disentangled representation learning, DeVAE, which utilizes hierarchical latent spaces with decreasing information bottlenecks across these spaces. The key innovation of our approach lies in connecting the hierarchical latent spaces through disentanglement-invariant transformations, allowing the sharing of disentanglement properties among spaces while maintaining an acceptable level of reconstruction performance. We demonstrate the effectiveness of DeVAE in achieving a balance between disentanglement and reconstruction through a series of experiments and ablation studies on dSprites and Shapes3D datasets. Code is available at https://github.com/erow/disentanglement_lib/tree/pytorch#devae.

Related papers

• Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration [64.84134880709625]
  We show that it is possible to perform domain adaptation via the noise space using diffusion models. In particular, by leveraging the unique property of how auxiliary conditional inputs influence the multi-step denoising process, we derive a meaningful diffusion loss. We present crucial strategies such as channel-shuffling layer and residual-swapping contrastive learning in the diffusion model.
  arXiv  Detail & Related papers  (2024-06-26T17:40:30Z)
• Distributional Reduction: Unifying Dimensionality Reduction and Clustering with Gromov-Wasserstein [56.62376364594194]
  Unsupervised learning aims to capture the underlying structure of potentially large and high-dimensional datasets. In this work, we revisit these approaches under the lens of optimal transport and exhibit relationships with the Gromov-Wasserstein problem. This unveils a new general framework, called distributional reduction, that recovers DR and clustering as special cases and allows addressing them jointly within a single optimization problem.
  arXiv  Detail & Related papers  (2024-02-03T19:00:19Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification in the Presence of Data Heterogeneity [60.791736094073]
  Communication overhead has become one of the major bottlenecks in the distributed training of deep neural networks. We propose a magnitude-driven sparsification scheme, which addresses the non-convergence issue of SIGNSGD. The proposed scheme is validated through experiments on Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets.
  arXiv  Detail & Related papers  (2023-02-19T17:42:35Z)
• Hub-VAE: Unsupervised Hub-based Regularization of Variational Autoencoders [11.252245456934348]
  We propose an unsupervised, data-driven regularization of the latent space with a mixture of hub-based priors and a hub-based contrastive loss. Our algorithm achieves superior cluster separability in the embedding space, and accurate data reconstruction and generation.
  arXiv  Detail & Related papers  (2022-11-18T19:12:15Z)
• Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning [112.69497636932955]
  Federated learning aims to train models across different clients without the sharing of data for privacy considerations. We study how data heterogeneity affects the representations of the globally aggregated models. We propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning.
  arXiv  Detail & Related papers  (2022-10-01T09:04:17Z)
• GCVAE: Generalized-Controllable Variational AutoEncoder [0.0]
  We present a framework to handle the trade-off between attaining extremely low reconstruction error and a high disentanglement score. We prove that maximizing information in the reconstruction network is equivalent to information during amortized inference.
  arXiv  Detail & Related papers  (2022-06-09T02:29:30Z)
• Learning disentangled representations with the Wasserstein Autoencoder [22.54887526392739]
  We propose TCWAE (Total Correlation Wasserstein Autoencoder) to penalize the total correlation in latent variables. We show that working in the WAE paradigm naturally enables the separation of the total-correlation term, thus providing disentanglement control over the learned representation. We further study the trade off between disentanglement and reconstruction on more-difficult data sets with unknown generative factors, where the flexibility of the WAE paradigm in the reconstruction term improves reconstructions.
  arXiv  Detail & Related papers  (2020-10-07T14:52:06Z)
• Representation Learning via Adversarially-Contrastive Optimal Transport [40.52344027750609]
  We set the problem within the context of contrastive representation learning. We propose a framework connecting Wasserstein GANs with a novel classifier. Our results demonstrate competitive performance against challenging baselines.
  arXiv  Detail & Related papers  (2020-07-11T19:46:18Z)
• Invariant Integration in Deep Convolutional Feature Space [77.99182201815763]
  We show how to incorporate prior knowledge to a deep neural network architecture in a principled manner. We report state-of-the-art performance on the Rotated-MNIST dataset.
  arXiv  Detail & Related papers  (2020-04-20T09:45:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.