LASERS: LAtent Space Encoding for Representations with Sparsity for Generative Modeling

• URL: http://arxiv.org/abs/2409.11184v1
• Date: Mon, 16 Sep 2024 08:20:58 GMT
• Title: LASERS: LAtent Space Encoding for Representations with Sparsity for Generative Modeling
• Authors: Xin Li, Anand Sarwate,
• Abstract summary: We show that our more latent space is more expressive and has leads to better representations than the Vector Quantization approach. Our results thus suggest that the true benefit of the VQ approach might not be from discretization of the latent space, but rather the lossy compression of the latent space.
• Score: 3.9426000822656224
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning compact and meaningful latent space representations has been shown to be very useful in generative modeling tasks for visual data. One particular example is applying Vector Quantization (VQ) in variational autoencoders (VQ-VAEs, VQ-GANs, etc.), which has demonstrated state-of-the-art performance in many modern generative modeling applications. Quantizing the latent space has been justified by the assumption that the data themselves are inherently discrete in the latent space (like pixel values). In this paper, we propose an alternative representation of the latent space by relaxing the structural assumption than the VQ formulation. Specifically, we assume that the latent space can be approximated by a union of subspaces model corresponding to a dictionary-based representation under a sparsity constraint. The dictionary is learned/updated during the training process. We apply this approach to look at two models: Dictionary Learning Variational Autoencoders (DL-VAEs) and DL-VAEs with Generative Adversarial Networks (DL-GANs). We show empirically that our more latent space is more expressive and has leads to better representations than the VQ approach in terms of reconstruction quality at the expense of a small computational overhead for the latent space computation. Our results thus suggest that the true benefit of the VQ approach might not be from discretization of the latent space, but rather the lossy compression of the latent space. We confirm this hypothesis by showing that our sparse representations also address the codebook collapse issue as found common in VQ-family models.

Related papers

• Addressing Representation Collapse in Vector Quantized Models with One Linear Layer [10.532262196027752]
  Vector Quantization (VQ) is a widely used method for converting continuous representations into discrete codes. VQ models are often hindered by the problem of representation collapse in the latent space. We propose textbfSimVQ, a novel method which re parameterizes the code vectors through a linear transformation layer based on a learnable latent basis.
  arXiv  Detail & Related papers  (2024-11-04T12:40:18Z)
• HyperVQ: MLR-based Vector Quantization in Hyperbolic Space [56.4245885674567]
  We study the use of hyperbolic spaces for vector quantization (HyperVQ) We show that hyperVQ performs comparably in reconstruction and generative tasks while outperforming VQ in discriminative tasks and learning a highly disentangled latent space.
  arXiv  Detail & Related papers  (2024-03-18T03:17:08Z)
• Vector Quantized Wasserstein Auto-Encoder [57.29764749855623]
  We study learning deep discrete representations from the generative viewpoint. We endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution. We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution.
  arXiv  Detail & Related papers  (2023-02-12T13:51:36Z)
• Barlow constrained optimization for Visual Question Answering [105.3372546782068]
  We propose a novel regularization for VQA models, Constrained Optimization using Barlow's theory (COB) Our model also aligns the joint space with the answer embedding space, where we consider the answer and image+question as two different views' of what in essence is the same semantic information. When built on the state-of-the-art GGE model, the resulting model improves VQA accuracy by 1.4% and 4% on the VQA-CP v2 and VQA v2 datasets respectively.
  arXiv  Detail & Related papers  (2022-03-07T21:27:40Z)
• Low-Rank Constraints for Fast Inference in Structured Models [110.38427965904266]
  This work demonstrates a simple approach to reduce the computational and memory complexity of a large class of structured models. Experiments with neural parameterized structured models for language modeling, polyphonic music modeling, unsupervised grammar induction, and video modeling show that our approach matches the accuracy of standard models at large state spaces.
  arXiv  Detail & Related papers  (2022-01-08T00:47:50Z)
• Discrete Auto-regressive Variational Attention Models for Text Modeling [53.38382932162732]
  Variational autoencoders (VAEs) have been widely applied for text modeling. They are troubled by two challenges: information underrepresentation and posterior collapse. We propose Discrete Auto-regressive Variational Attention Model (DAVAM) to address the challenges.
  arXiv  Detail & Related papers  (2021-06-16T06:36:26Z)
• IntroVAC: Introspective Variational Classifiers for Learning Interpretable Latent Subspaces [6.574517227976925]
  IntroVAC learns interpretable latent subspaces by exploiting information from an additional label. We show that IntroVAC is able to learn meaningful directions in the latent space enabling fine manipulation of image attributes.
  arXiv  Detail & Related papers  (2020-08-03T10:21:41Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)
• Depthwise Discrete Representation Learning [2.728575246952532]
  Recent advancements in learning Discrete Representations have led to state of art results in tasks that involve Language, Audio and Vision. Some latent factors such as words, phonemes and shapes are better represented by discrete latent variables as opposed to continuous. Vector Quantized Variational Autoencoders (VQVAE) have produced remarkable results in multiple domains.
  arXiv  Detail & Related papers  (2020-04-11T18:57:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.