DynamicVAE: Decoupling Reconstruction Error and Disentangled Representation Learning

• URL: http://arxiv.org/abs/2009.06795v2
• Date: Wed, 30 Sep 2020 22:11:07 GMT
• Title: DynamicVAE: Decoupling Reconstruction Error and Disentangled Representation Learning
• Authors: Huajie Shao, Haohong Lin, Qinmin Yang, Shuochao Yao, Han Zhao, Tarek Abdelzaher
• Abstract summary: This paper challenges the common assumption that the weight $beta$, in $beta$-VAE, should be larger than $1$ in order to effectively disentangle latent factors. We demonstrate that $beta$-VAE, with $beta 1$, can not only attain good disentanglement but also significantly improve reconstruction accuracy via dynamic control.
• Score: 15.317044259237043
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper challenges the common assumption that the weight $\beta$, in $\beta$-VAE, should be larger than $1$ in order to effectively disentangle latent factors. We demonstrate that $\beta$-VAE, with $\beta < 1$, can not only attain good disentanglement but also significantly improve reconstruction accuracy via dynamic control. The paper removes the inherent trade-off between reconstruction accuracy and disentanglement for $\beta$-VAE. Existing methods, such as $\beta$-VAE and FactorVAE, assign a large weight to the KL-divergence term in the objective function, leading to high reconstruction errors for the sake of better disentanglement. To mitigate this problem, a ControlVAE has recently been developed that dynamically tunes the KL-divergence weight in an attempt to control the trade-off to more a favorable point. However, ControlVAE fails to eliminate the conflict between the need for a large $\beta$ (for disentanglement) and the need for a small $\beta$. Instead, we propose DynamicVAE that maintains a different $\beta$ at different stages of training, thereby decoupling disentanglement and reconstruction accuracy. In order to evolve the weight, $\beta$, along a trajectory that enables such decoupling, DynamicVAE leverages a modified incremental PI (proportional-integral) controller, and employs a moving average as well as a hybrid annealing method to evolve the value of KL-divergence smoothly in a tightly controlled fashion. We theoretically prove the stability of the proposed approach. Evaluation results on three benchmark datasets demonstrate that DynamicVAE significantly improves the reconstruction accuracy while achieving disentanglement comparable to the best of existing methods. The results verify that our method can separate disentangled representation learning and reconstruction, removing the inherent tension between the two.

Related papers

• Towards aerodynamic surrogate modeling based on $β$-variational autoencoders [0.0]
  Surrogate models that combine dimensionality reduction and regression techniques are essential to reduce the need for costly high-fidelity computational fluid dynamics data. We propose a surrogate model based on latent space regression to predict pressure distributions on a transonic wing given the flight conditions: Mach number and angle of attack.
  arXiv  Detail & Related papers  (2024-08-09T09:43:10Z)
• Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption [60.958746600254884]
  This study tackles the challenges of adversarial corruption in model-based reinforcement learning (RL) We introduce an algorithm called corruption-robust optimistic MLE (CR-OMLE), which leverages total-variation (TV)-based information ratios as uncertainty weights for MLE. We extend our weighting technique to the offline setting, and propose an algorithm named corruption-robust pessimistic MLE (CR-PMLE)
  arXiv  Detail & Related papers  (2024-02-14T07:27:30Z)
• Uncertainty Guided Adaptive Warping for Robust and Efficient Stereo Matching [77.133400999703]
  Correlation based stereo matching has achieved outstanding performance. Current methods with a fixed model do not work uniformly well across various datasets. This paper proposes a new perspective to dynamically calculate correlation for robust stereo matching.
  arXiv  Detail & Related papers  (2023-07-26T09:47:37Z)
• Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve [29.86440019821837]
  Variational autoencoders (VAEs) are powerful tools for learning latent representations of data used in a wide range of applications. In this paper, we introduce Multi-Rate VAE, a computationally efficient framework for learning optimal parameters corresponding to various $beta$ in a single training run.
  arXiv  Detail & Related papers  (2022-12-07T19:02:34Z)
• Training \beta-VAE by Aggregating a Learned Gaussian Posterior with a Decoupled Decoder [0.553073476964056]
  Current practices in VAE training often result in a trade-off between the reconstruction fidelity and the continuity$/$disentanglement of the latent space. We present intuitions and a careful analysis of the antagonistic mechanism of the two losses, and propose a simple yet effective two-stage method for training a VAE. We evaluate the method using a medical dataset intended for 3D skull reconstruction and shape completion, and the results indicate promising generative capabilities of the VAE trained using the proposed method.
  arXiv  Detail & Related papers  (2022-09-29T13:49:57Z)
• A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning [113.75991721607174]
  We introduce an interventional prediction module to estimate the probability of two estimated $hatz_i, hatz_j$ belonging to the same environment. We empirically show that $hatZ$ estimated by our method enjoy less redundant information than previous methods.
  arXiv  Detail & Related papers  (2022-06-09T15:01:36Z)
• Robust Online Control with Model Misspecification [96.23493624553998]
  We study online control of an unknown nonlinear dynamical system with model misspecification. Our study focuses on robustness, which measures how much deviation from the assumed linear approximation can be tolerated.
  arXiv  Detail & Related papers  (2021-07-16T07:04:35Z)
• Demystifying Inductive Biases for $\beta$-VAE Based Architectures [19.53632220171481]
  We shed light on the inductive bias responsible for the success of VAE-based architectures. We show that in classical datasets the structure of variance, induced by the generating factors, is conveniently aligned with the latent directions fostered by the VAE objective.
  arXiv  Detail & Related papers  (2021-02-12T23:57:20Z)
• ControlVAE: Tuning, Analytical Properties, and Performance Analysis [14.272917020105147]
  ControlVAE is a new variational autoencoder framework. It stabilizes the KL-divergence of VAE models to a specified value. It can achieve a good trade-off between reconstruction quality and KL-divergence.
  arXiv  Detail & Related papers  (2020-10-31T12:32:39Z)
• Understanding the Difficulty of Training Transformers [120.99980924577787]
  We show that unbalanced gradients are not the root cause of the instability of training. We propose Admin to stabilize the early stage's training and unleash its full potential in the late stage.
  arXiv  Detail & Related papers  (2020-04-17T13:59:07Z)
• Momentum Improves Normalized SGD [51.27183254738711]
  We show that adding momentum provably removes the need for large batch sizes on objectives. We show that our method is effective when employed on popular large scale tasks such as ResNet-50 and BERT pretraining.
  arXiv  Detail & Related papers  (2020-02-09T07:00:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.