PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders

• URL: http://arxiv.org/abs/2007.06503v1
• Date: Mon, 13 Jul 2020 16:56:00 GMT
• Title: PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders
• Authors: Yanjun Li, Shujian Yu, Jose C. Principe, Xiaolin Li, and Dapeng Wu
• Abstract summary: We first propose a novel learning objective, termed the principle-of-relevant-information variational autoencoder (PRI-VAE) We present an information-theoretic perspective to analyze existing VAE models by inspecting the evolution of some critical information-theoretic quantities. Empirical results also demonstrate the effectiveness of PRI-VAE on four benchmark data sets.
• Score: 21.702801479284986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although substantial efforts have been made to learn disentangled representations under the variational autoencoder (VAE) framework, the fundamental properties to the dynamics of learning of most VAE models still remain unknown and under-investigated. In this work, we first propose a novel learning objective, termed the principle-of-relevant-information variational autoencoder (PRI-VAE), to learn disentangled representations. We then present an information-theoretic perspective to analyze existing VAE models by inspecting the evolution of some critical information-theoretic quantities across training epochs. Our observations unveil some fundamental properties associated with VAEs. Empirical results also demonstrate the effectiveness of PRI-VAE on four benchmark data sets.

Related papers

• Prior Learning in Introspective VAEs [24.271671383057598]
  Variational Autoencoders (VAEs) are a popular framework for unsupervised learning and data generation. In this study, we focus on the Soft-IntroVAE and investigate the implication of incorporating a multimodal and learnable prior into this framework.
  arXiv  Detail & Related papers  (2024-08-25T10:54:25Z)
• Silkie: Preference Distillation for Large Visual Language Models [56.10697821410489]
  This paper explores preference distillation for large vision language models (LVLMs) We first build a vision-language feedback dataset utilizing AI annotation. We adopt GPT-4V to assess the generated outputs regarding helpfulness, visual faithfulness, and ethical considerations. The resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME benchmark regarding the perception and cognition capabilities.
  arXiv  Detail & Related papers  (2023-12-17T09:44:27Z)
• Improving VAE-based Representation Learning [26.47244578124654]
  We study what properties are required for good representations and how different VAE structure choices could affect the learned properties. We show that by using a decoder that prefers to learn local features, the remaining global features can be well captured by the latent.
  arXiv  Detail & Related papers  (2022-05-28T23:00:18Z)
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning [56.20123080771364]
  We develop a model-agnostic Contrastive-Curiosity-Driven Learning Framework (CCLF) for reinforcement learning. CCLF fully exploit sample importance and improve learning efficiency in a self-supervised manner. We evaluate this approach on the DeepMind Control Suite, Atari, and MiniGrid benchmarks.
  arXiv  Detail & Related papers  (2022-05-02T14:42:05Z)
• Covariate-informed Representation Learning with Samplewise Optimal Identifiable Variational Autoencoders [15.254297587065595]
  Recently proposed identifiable variational autoencoder (iVAE) provides a promising approach for learning latent independent components of the data. We develop a new approach, co-informed identifiable VAE (CI-iVAE) In doing so, the objective function enforces the inverse relation, and learned representation contains more information of observations.
  arXiv  Detail & Related papers  (2022-02-09T00:18:33Z)
• Leveraging Intrinsic Gradient Information for Machine Learning Model Training [4.682734815593623]
  derivatives of target variables with respect to inputs can be leveraged to improve the accuracy of differentiable machine learning models. Four key ideas are explored: (1) Improving the predictive accuracy of linear regression models and feed-forward neural networks (NNs); (2) Using the difference between the performance of feedforward NNs trained with and without gradient information to tune NN complexity; and (4) Using gradient information to improve generative image models.
  arXiv  Detail & Related papers  (2021-11-30T20:50:45Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• InteL-VAEs: Adding Inductive Biases to Variational Auto-Encoders via Intermediary Latents [60.785317191131284]
  We introduce a simple and effective method for learning VAEs with controllable biases by using an intermediary set of latent variables. In particular, it allows us to impose desired properties like sparsity or clustering on learned representations. We show that this, in turn, allows InteL-VAEs to learn both better generative models and representations.
  arXiv  Detail & Related papers  (2021-06-25T16:34:05Z)
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks [99.44881722969046]
  We introduce a conceptually simple yet effective model for self-supervised representation learning with graph data. We optimize an innovative feature-level objective inspired by classical Canonical Correlation Analysis. Our method performs competitively on seven public graph datasets.
  arXiv  Detail & Related papers  (2021-06-23T15:55:47Z)
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study [81.11161697133095]
  We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
  arXiv  Detail & Related papers  (2020-01-12T04:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.