Robust Vector Quantized-Variational Autoencoder
- URL: http://arxiv.org/abs/2202.01987v1
- Date: Fri, 4 Feb 2022 05:51:15 GMT
- Title: Robust Vector Quantized-Variational Autoencoder
- Authors: Chieh-Hsin Lai, Dongmian Zou, Gilad Lerman
- Abstract summary: We propose a robust generative model based on Vector Quantized-Variational AutoEncoder (VQ-VAE)
In order to achieve robustness, RVQ-VAE uses two separate codebooks for the inliers and outliers.
We experimentally demonstrate that RVQ-VAE is able to generate examples from inliers even if a large portion of the training data points are corrupted.
- Score: 13.664682865991255
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Image generative models can learn the distributions of the training data and
consequently generate examples by sampling from these distributions. However,
when the training dataset is corrupted with outliers, generative models will
likely produce examples that are also similar to the outliers. In fact, a small
portion of outliers may induce state-of-the-art generative models, such as
Vector Quantized-Variational AutoEncoder (VQ-VAE), to learn a significant mode
from the outliers. To mitigate this problem, we propose a robust generative
model based on VQ-VAE, which we name Robust VQ-VAE (RVQ-VAE). In order to
achieve robustness, RVQ-VAE uses two separate codebooks for the inliers and
outliers. To ensure the codebooks embed the correct components, we iteratively
update the sets of inliers and outliers during each training epoch. To ensure
that the encoded data points are matched to the correct codebooks, we quantize
using a weighted Euclidean distance, whose weights are determined by
directional variances of the codebooks. Both codebooks, together with the
encoder and decoder, are trained jointly according to the reconstruction loss
and the quantization loss. We experimentally demonstrate that RVQ-VAE is able
to generate examples from inliers even if a large portion of the training data
points are corrupted.
Related papers
- EdVAE: Mitigating Codebook Collapse with Evidential Discrete Variational Autoencoders [11.086500036180222]
Codebook collapse is a common problem in training deep generative models with discrete representation spaces.
We propose a novel way to incorporate evidential deep learning (EDL) instead of softmax to combat the codebook collapse problem of dVAE.
arXiv Detail & Related papers (2023-10-09T13:39:26Z) - Online Clustered Codebook [100.1650001618827]
We present a simple alternative method for online codebook learning, Clustering VQ-VAE (CVQ-VAE)
Our approach selects encoded features as anchors to update the dead'' codevectors, while optimising the codebooks which are alive via the original loss.
Our CVQ-VAE can be easily integrated into the existing models with just a few lines of code.
arXiv Detail & Related papers (2023-07-27T18:31:04Z) - Enhancing Multiple Reliability Measures via Nuisance-extended
Information Bottleneck [77.37409441129995]
In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition.
We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training.
We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
arXiv Detail & Related papers (2023-03-24T16:03:21Z) - Vector Quantized Wasserstein Auto-Encoder [57.29764749855623]
We study learning deep discrete representations from the generative viewpoint.
We endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution.
We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution.
arXiv Detail & Related papers (2023-02-12T13:51:36Z) - Few-Shot Non-Parametric Learning with Deep Latent Variable Model [50.746273235463754]
We propose Non-Parametric learning by Compression with Latent Variables (NPC-LV)
NPC-LV is a learning framework for any dataset with abundant unlabeled data but very few labeled ones.
We show that NPC-LV outperforms supervised methods on all three datasets on image classification in low data regime.
arXiv Detail & Related papers (2022-06-23T09:35:03Z) - Diffusion bridges vector quantized Variational AutoEncoders [0.0]
We show that our model is competitive with the autoregressive prior on the mini-Imagenet dataset.
Our framework also extends the standard VQ-VAE and enables end-to-end training.
arXiv Detail & Related papers (2022-02-10T08:38:12Z) - Autoencoding Variational Autoencoder [56.05008520271406]
We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency.
We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
arXiv Detail & Related papers (2020-12-07T14:16:14Z) - Robust Variational Autoencoder for Tabular Data with Beta Divergence [0.0]
We propose a robust variational autoencoder with mixed categorical and continuous features.
Our results on the anomaly detection application for network traffic datasets demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2020-06-15T08:09:34Z) - Robust Training of Vector Quantized Bottleneck Models [21.540133031071438]
We demonstrate methods for reliable and efficient training of discrete representation using Vector-Quantized Variational Auto-Encoder models (VQ-VAEs)
For unsupervised representation learning, they became viable alternatives to continuous latent variable models such as the Variational Auto-Encoder (VAE)
arXiv Detail & Related papers (2020-05-18T08:23:41Z) - Unshuffling Data for Improved Generalization [65.57124325257409]
Generalization beyond the training distribution is a core challenge in machine learning.
We show that partitioning the data into well-chosen, non-i.i.d. subsets treated as multiple training environments can guide the learning of models with better out-of-distribution generalization.
arXiv Detail & Related papers (2020-02-27T03:07:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.