Depthwise Discrete Representation Learning
- URL: http://arxiv.org/abs/2004.05462v1
- Date: Sat, 11 Apr 2020 18:57:13 GMT
- Title: Depthwise Discrete Representation Learning
- Authors: Iordanis Fostiropoulos
- Abstract summary: Recent advancements in learning Discrete Representations have led to state of art results in tasks that involve Language, Audio and Vision.
Some latent factors such as words, phonemes and shapes are better represented by discrete latent variables as opposed to continuous.
Vector Quantized Variational Autoencoders (VQVAE) have produced remarkable results in multiple domains.
- Score: 2.728575246952532
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent advancements in learning Discrete Representations as opposed to
continuous ones have led to state of art results in tasks that involve
Language, Audio and Vision. Some latent factors such as words, phonemes and
shapes are better represented by discrete latent variables as opposed to
continuous. Vector Quantized Variational Autoencoders (VQVAE) have produced
remarkable results in multiple domains. VQVAE learns a prior distribution $z_e$
along with its mapping to a discrete number of $K$ vectors (Vector
Quantization). We propose applying VQ along the feature axis. We hypothesize
that by doing so, we are learning a mapping between the codebook vectors and
the marginal distribution of the prior feature space. Our approach leads to
33\% improvement as compared to prevous discrete models and has similar
performance to state of the art auto-regressive models (e.g. PixelSNAIL). We
evaluate our approach on a static prior using an artificial toy dataset
(blobs). We further evaluate our approach on benchmarks for CIFAR-10 and
ImageNet.
Related papers
- LASERS: LAtent Space Encoding for Representations with Sparsity for Generative Modeling [3.9426000822656224]
We show that our more latent space is more expressive and has leads to better representations than the Vector Quantization approach.
Our results thus suggest that the true benefit of the VQ approach might not be from discretization of the latent space, but rather the lossy compression of the latent space.
arXiv Detail & Related papers (2024-09-16T08:20:58Z) - LL-VQ-VAE: Learnable Lattice Vector-Quantization For Efficient
Representations [0.0]
We introduce learnable lattice vector quantization and demonstrate its effectiveness for learning discrete representations.
Our method, termed LL-VQ-VAE, replaces the vector quantization layer in VQ-VAE with lattice-based discretization.
Compared to VQ-VAE, our method obtains lower reconstruction errors under the same training conditions, trains in a fraction of the time, and with a constant number of parameters.
arXiv Detail & Related papers (2023-10-13T20:03:18Z) - Value-Distributional Model-Based Reinforcement Learning [59.758009422067]
Quantifying uncertainty about a policy's long-term performance is important to solve sequential decision-making tasks.
We study the problem from a model-based Bayesian reinforcement learning perspective.
We propose Epistemic Quantile-Regression (EQR), a model-based algorithm that learns a value distribution function.
arXiv Detail & Related papers (2023-08-12T14:59:19Z) - Discrete Graph Auto-Encoder [52.50288418639075]
We introduce a new framework named Discrete Graph Auto-Encoder (DGAE)
We first use a permutation-equivariant auto-encoder to convert graphs into sets of discrete latent node representations.
In the second step, we sort the sets of discrete latent representations and learn their distribution with a specifically designed auto-regressive model.
arXiv Detail & Related papers (2023-06-13T12:40:39Z) - Vector Quantized Wasserstein Auto-Encoder [57.29764749855623]
We study learning deep discrete representations from the generative viewpoint.
We endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution.
We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution.
arXiv Detail & Related papers (2023-02-12T13:51:36Z) - Diffusion bridges vector quantized Variational AutoEncoders [0.0]
We show that our model is competitive with the autoregressive prior on the mini-Imagenet dataset.
Our framework also extends the standard VQ-VAE and enables end-to-end training.
arXiv Detail & Related papers (2022-02-10T08:38:12Z) - Adaptive Discrete Communication Bottlenecks with Dynamic Vector
Quantization [76.68866368409216]
We propose learning to dynamically select discretization tightness conditioned on inputs.
We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks.
arXiv Detail & Related papers (2022-02-02T23:54:26Z) - Revisiting Contrastive Methods for Unsupervised Learning of Visual
Representations [78.12377360145078]
Contrastive self-supervised learning has outperformed supervised pretraining on many downstream tasks like segmentation and object detection.
In this paper, we first study how biases in the dataset affect existing methods.
We show that current contrastive approaches work surprisingly well across: (i) object- versus scene-centric, (ii) uniform versus long-tailed and (iii) general versus domain-specific datasets.
arXiv Detail & Related papers (2021-06-10T17:59:13Z) - How to Design Sample and Computationally Efficient VQA Models [53.65668097847456]
We find that representing the text as probabilistic programs and images as object-level scene graphs best satisfy these desiderata.
We extend existing models to leverage these soft programs and scene graphs to train on question answer pairs in an end-to-end manner.
arXiv Detail & Related papers (2021-03-22T01:48:16Z) - Robust Training of Vector Quantized Bottleneck Models [21.540133031071438]
We demonstrate methods for reliable and efficient training of discrete representation using Vector-Quantized Variational Auto-Encoder models (VQ-VAEs)
For unsupervised representation learning, they became viable alternatives to continuous latent variable models such as the Variational Auto-Encoder (VAE)
arXiv Detail & Related papers (2020-05-18T08:23:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.