The Neglected Sibling: Isotropic Gaussian Posterior for VAE

• URL: http://arxiv.org/abs/2110.07383v1
• Date: Thu, 14 Oct 2021 14:12:52 GMT
• Title: The Neglected Sibling: Isotropic Gaussian Posterior for VAE
• Abstract summary: We propose a simple modification to Variational Autoencoders (VAEs) by using an Isotropic Gaussian Posterior (IGP) This model avoids the sub-optimal behavior of VAEs related to inactive dimensions in the representation space.
• Score: 5.377898309773304
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deep generative models have been widely used in several areas of NLP, and various techniques have been proposed to augment them or address their training challenges. In this paper, we propose a simple modification to Variational Autoencoders (VAEs) by using an Isotropic Gaussian Posterior (IGP) that allows for better utilisation of their latent representation space. This model avoids the sub-optimal behavior of VAEs related to inactive dimensions in the representation space. We provide both theoretical analysis, and empirical evidence on various datasets and tasks that show IGP leads to consistent improvement on several quantitative and qualitative grounds, from downstream task performance and sample efficiency to robustness. Additionally, we give insights about the representational properties encouraged by IGP and also show that its gain generalises to image domain as well.

Related papers

• Generalizable Insights for Graph Transformers in Theory and Practice [10.811483927795921]
  Graph Transformers (GTs) have shown strong empirical performance.<n>Current architectures vary widely in their use of attention mechanisms, positional embeddings (PEs), and expressivity.<n>Here, we propose a GT architecture using standard attention that incorporates many advancements for GTs from recent years.
  arXiv  Detail & Related papers  (2025-11-11T09:30:22Z)
• PIGPVAE: Physics-Informed Gaussian Process Variational Autoencoders [42.8983261737774]
  We propose a novel generative model that learns from limited data by incorporating physical constraints to enhance performance.<n>We extend the VAE architecture by incorporating physical models in the generative process, enabling it to capture underlying dynamics more effectively.<n>We demonstrate that PIGPVAE can produce realistic samples beyond the observed distribution, highlighting its robustness and usefulness under distribution shifts.
  arXiv  Detail & Related papers  (2025-05-25T21:12:01Z)
• Neighbour-Driven Gaussian Process Variational Autoencoders for Scalable Structured Latent Modelling [14.358070928996069]
  Gaussian Process (GP) Variational Autoencoders (VAEs) extend standard VAEs by replacing the fully factorised Gaussian prior with a GP prior.<n> performing exact GP inference in large-scale GPVAEs is computationally prohibitive, often forcing existing approaches to rely on restrictive kernel assumptions.<n>We propose a neighbour-driven approximation strategy that exploits local adjacencies in the latent space to achieve scalable GPVAE inference.
  arXiv  Detail & Related papers  (2025-05-22T10:07:33Z)
• Multi-View Oriented GPLVM: Expressiveness and Efficiency [8.459922325396155]
  We introduce a new duality between the spectral density and the kernel function. We derive a generic and expressive kernel termed Next-Gen Spectral Mixture (NG-SM) for MV-GPLVMs. Our proposed method consistently outperforms state-of-the-art models in learning meaningful latent representations.
  arXiv  Detail & Related papers  (2025-02-12T09:49:25Z)
• Exploring Representation-Aligned Latent Space for Better Generation [86.45670422239317]
  We introduce ReaLS, which integrates semantic priors to improve generation performance.<n>We show that fundamental DiT and SiT trained on ReaLS can achieve a 15% improvement in FID metric.<n>The enhanced semantic latent space enables more perceptual downstream tasks, such as segmentation and depth estimation.
  arXiv  Detail & Related papers  (2025-02-01T07:42:12Z)
• $α$-TCVAE: On the relationship between Disentanglement and Diversity [21.811889512977924]
  In this work, we introduce $alpha$-TCVAE, a variational autoencoder optimized using a novel total correlation (TC) lower bound. We present quantitative analyses that support the idea that disentangled representations lead to better generative capabilities and diversity. Our results demonstrate that $alpha$-TCVAE consistently learns more disentangled representations than baselines and generates more diverse observations.
  arXiv  Detail & Related papers  (2024-11-01T13:50:06Z)
• Theoretical Convergence Guarantees for Variational Autoencoders [2.8167997311962942]
  Variational Autoencoders (VAE) are popular generative models used to sample from complex data distributions. This paper aims to bridge that gap by providing non-asymptotic convergence guarantees for VAE trained using both Gradient Descent and Adam algorithms. Our theoretical analysis applies to both Linear VAE and Deep Gaussian VAE, as well as several VAE variants, including $beta$-VAE and IWAE.
  arXiv  Detail & Related papers  (2024-10-22T07:12:38Z)
• QT-DoG: Quantization-aware Training for Domain Generalization [58.439816306817306]
  We propose Quantization-aware Training for Domain Generalization (QT-DoG) QT-DoG exploits quantization as an implicit regularizer by inducing noise in model weights. We demonstrate that QT-DoG generalizes across various datasets, architectures, and quantization algorithms.
  arXiv  Detail & Related papers  (2024-10-08T13:21:48Z)
• Spherical Inducing Features for Orthogonally-Decoupled Gaussian Processes [7.4468224549568705]
  Gaussian processes (GPs) are often compared unfavorably to deep neural networks (NNs) for lacking the ability to learn representations. Recent efforts to bridge the gap between GPs and deep NNs have yielded a new class of inter-domain variational GPs in which the inducing variables correspond to hidden units of a feedforward NN.
  arXiv  Detail & Related papers  (2023-04-27T09:00:02Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• A Generalized EigenGame with Extensions to Multiview Representation Learning [0.28647133890966997]
  Generalized Eigenvalue Problems (GEPs) encompass a range of interesting dimensionality reduction methods. We develop an approach to solving GEPs in which all constraints are softly enforced by Lagrange multipliers. We show that our approaches share much of the theoretical grounding of the previous Hebbian and game theoretic approaches for the linear case. We demonstrate the effectiveness of our method for solving GEPs in the setting of canonical multiview datasets.
  arXiv  Detail & Related papers  (2022-11-21T10:11:13Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Deep Variational Models for Collaborative Filtering-based Recommender Systems [63.995130144110156]
  Deep learning provides accurate collaborative filtering models to improve recommender system results. Our proposed models apply the variational concept to injectity in the latent space of the deep architecture. Results show the superiority of the proposed approach in scenarios where the variational enrichment exceeds the injected noise effect.
  arXiv  Detail & Related papers  (2021-07-27T08:59:39Z)
• High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning [119.91679702854499]
  We introduce a method based on deep metric learning to perform Bayesian optimisation over high-dimensional, structured input spaces. We achieve such an inductive bias using just 1% of the available labelled data. As an empirical contribution, we present state-of-the-art results on real-world high-dimensional black-box optimisation problems.
  arXiv  Detail & Related papers  (2021-06-07T13:35:47Z)
• Latent Map Gaussian Processes for Mixed Variable Metamodeling [0.0]
  We introduce latent map Gaussian processes (LMGPs) that inherit the attractive properties of GPs but are also applicable to mixed data. We show that LMGPs can handle variable-length inputs and provide insights into how qualitative inputs affect the response or interact with each other. We also provide a neural network interpretation of LMGPs and study the effect of prior latent representations on their performance.
  arXiv  Detail & Related papers  (2021-02-07T22:21:53Z)
• Self-Guided Adaptation: Progressive Representation Alignment for Domain Adaptive Object Detection [86.69077525494106]
  Unsupervised domain adaptation (UDA) has achieved unprecedented success in improving the cross-domain robustness of object detection models. Existing UDA methods largely ignore the instantaneous data distribution during model learning, which could deteriorate the feature representation given large domain shift. We propose a Self-Guided Adaptation (SGA) model, target at aligning feature representation and transferring object detection models across domains.
  arXiv  Detail & Related papers  (2020-03-19T13:30:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.