Variational Mutual Information Maximization Framework for VAE Latent Codes with Continuous and Discrete Priors

• URL: http://arxiv.org/abs/2006.02227v1
• Date: Tue, 2 Jun 2020 09:05:51 GMT
• Title: Variational Mutual Information Maximization Framework for VAE Latent Codes with Continuous and Discrete Priors
• Authors: Andriy Serdega, Dae-Shik Kim
• Abstract summary: Variational Autoencoder (VAE) is a scalable method for learning directed latent variable models of complex data. We propose Variational Mutual Information Maximization Framework for VAE to address this issue.
• Score: 5.317548969642376
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning interpretable and disentangled representations of data is a key topic in machine learning research. Variational Autoencoder (VAE) is a scalable method for learning directed latent variable models of complex data. It employs a clear and interpretable objective that can be easily optimized. However, this objective does not provide an explicit measure for the quality of latent variable representations which may result in their poor quality. We propose Variational Mutual Information Maximization Framework for VAE to address this issue. In comparison to other methods, it provides an explicit objective that maximizes lower bound on mutual information between latent codes and observations. The objective acts as a regularizer that forces VAE to not ignore the latent variable and allows one to select particular components of it to be most informative with respect to the observations. On top of that, the proposed framework provides a way to evaluate mutual information between latent codes and observations for a fixed VAE model. We have conducted our experiments on VAE models with Gaussian and joint Gaussian and discrete latent variables. Our results illustrate that the proposed approach strengthens relationships between latent codes and observations and improves learned representations.

Related papers

• Self-Supervised Representation Learning with Meta Comprehensive Regularization [11.387994024747842]
  We introduce a module called CompMod with Meta Comprehensive Regularization (MCR), embedded into existing self-supervised frameworks. We update our proposed model through a bi-level optimization mechanism, enabling it to capture comprehensive features. We provide theoretical support for our proposed method from information theory and causal counterfactual perspective.
  arXiv  Detail & Related papers  (2024-03-03T15:53:48Z)
• RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling [79.56442336234221]
  We introduce RegaVAE, a retrieval-augmented language model built upon the variational auto-encoder (VAE) It encodes the text corpus into a latent space, capturing current and future information from both source and target text. Experimental results on various datasets demonstrate significant improvements in text generation quality and hallucination removal.
  arXiv  Detail & Related papers  (2023-10-16T16:42:01Z)
• DCID: Deep Canonical Information Decomposition [84.59396326810085]
  We consider the problem of identifying the signal shared between two one-dimensional target variables. We propose ICM, an evaluation metric which can be used in the presence of ground-truth labels. We also propose Deep Canonical Information Decomposition (DCID) - a simple, yet effective approach for learning the shared variables.
  arXiv  Detail & Related papers  (2023-06-27T16:59:06Z)
• Vector Quantized Wasserstein Auto-Encoder [57.29764749855623]
  We study learning deep discrete representations from the generative viewpoint. We endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution. We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution.
  arXiv  Detail & Related papers  (2023-02-12T13:51:36Z)
• Covariate-informed Representation Learning with Samplewise Optimal Identifiable Variational Autoencoders [15.254297587065595]
  Recently proposed identifiable variational autoencoder (iVAE) provides a promising approach for learning latent independent components of the data. We develop a new approach, co-informed identifiable VAE (CI-iVAE) In doing so, the objective function enforces the inverse relation, and learned representation contains more information of observations.
  arXiv  Detail & Related papers  (2022-02-09T00:18:33Z)
• Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization [76.68866368409216]
  We propose learning to dynamically select discretization tightness conditioned on inputs. We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks.
  arXiv  Detail & Related papers  (2022-02-02T23:54:26Z)
• InteL-VAEs: Adding Inductive Biases to Variational Auto-Encoders via Intermediary Latents [60.785317191131284]
  We introduce a simple and effective method for learning VAEs with controllable biases by using an intermediary set of latent variables. In particular, it allows us to impose desired properties like sparsity or clustering on learned representations. We show that this, in turn, allows InteL-VAEs to learn both better generative models and representations.
  arXiv  Detail & Related papers  (2021-06-25T16:34:05Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• VMI-VAE: Variational Mutual Information Maximization Framework for VAE With Discrete and Continuous Priors [5.317548969642376]
  Variational Autoencoder is a scalable method for learning latent variable models of complex data. We propose a Variational Mutual Information Maximization Framework for VAE to address this issue.
  arXiv  Detail & Related papers  (2020-05-28T12:44:23Z)
• Deterministic Decoding for Discrete Data in Variational Autoencoders [5.254093731341154]
  We study a VAE model with a deterministic decoder (DD-VAE) for sequential data that selects the highest-scoring tokens instead of sampling. We demonstrate the performance of DD-VAE on multiple datasets, including molecular generation and optimization problems.
  arXiv  Detail & Related papers  (2020-03-04T16:36:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.