Variational Autoencoders with Riemannian Brownian Motion Priors

• URL: http://arxiv.org/abs/2002.05227v3
• Date: Fri, 7 Aug 2020 16:01:28 GMT
• Title: Variational Autoencoders with Riemannian Brownian Motion Priors
• Authors: Dimitris Kalatzis, David Eklund, Georgios Arvanitidis, S{\o}ren Hauberg
• Abstract summary: Variational Autoencoders (VAEs) represent the given data in a low-dimensional latent space, which is generally assumed to be Euclidean. Recent work has shown that this prior has a detrimental effect on model capacity, leading to subpar performance. We propose an efficient inference scheme that does not rely on the unknown normalizing factor of this prior.
• Score: 4.8461049669050915
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Variational Autoencoders (VAEs) represent the given data in a low-dimensional latent space, which is generally assumed to be Euclidean. This assumption naturally leads to the common choice of a standard Gaussian prior over continuous latent variables. Recent work has, however, shown that this prior has a detrimental effect on model capacity, leading to subpar performance. We propose that the Euclidean assumption lies at the heart of this failure mode. To counter this, we assume a Riemannian structure over the latent space, which constitutes a more principled geometric view of the latent codes, and replace the standard Gaussian prior with a Riemannian Brownian motion prior. We propose an efficient inference scheme that does not rely on the unknown normalizing factor of this prior. Finally, we demonstrate that this prior significantly increases model capacity using only one additional scalar parameter.

Related papers

• Preconditioned Langevin Dynamics with Score-Based Generative Models for Infinite-Dimensional Linear Bayesian Inverse Problems [4.2223436389469144]
  Langevin dynamics driven by score-based generative models (SGMs) acting as priors, formulated directly in function space.<n>We derive, for the first time, error estimates that explicitly depend on the approximation error of the score.<n>As a consequence, we obtain sufficient conditions for global convergence in Kullback-Leibler divergence on the underlying function space.
  arXiv  Detail & Related papers  (2025-05-23T18:12:04Z)
• A Non-negative VAE:the Generalized Gamma Belief Network [49.970917207211556]
  The gamma belief network (GBN) has demonstrated its potential for uncovering multi-layer interpretable latent representations in text data. We introduce the generalized gamma belief network (Generalized GBN) in this paper, which extends the original linear generative model to a more expressive non-linear generative model. We also propose an upward-downward Weibull inference network to approximate the posterior distribution of the latent variables.
  arXiv  Detail & Related papers  (2024-08-06T18:18:37Z)
• von Mises Quasi-Processes for Bayesian Circular Regression [57.88921637944379]
  We explore a family of expressive and interpretable distributions over circle-valued random functions. The resulting probability model has connections with continuous spin models in statistical physics. For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Markov Chain Monte Carlo sampling.
  arXiv  Detail & Related papers  (2024-06-19T01:57:21Z)
• Convex Parameter Estimation of Perturbed Multivariate Generalized Gaussian Distributions [18.95928707619676]
  We propose a convex formulation with well-established properties for MGGD parameters. The proposed framework is flexible as it combines a variety of regularizations for the precision matrix, the mean and perturbations. Experiments show a more accurate precision and covariance matrix estimation with similar performance for the mean vector parameter.
  arXiv  Detail & Related papers  (2023-12-12T18:08:04Z)
• Curvature-Independent Last-Iterate Convergence for Games on Riemannian Manifolds [77.4346324549323]
  We show that a step size agnostic to the curvature of the manifold achieves a curvature-independent and linear last-iterate convergence rate. To the best of our knowledge, the possibility of curvature-independent rates and/or last-iterate convergence has not been considered before.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• The Choice of Noninformative Priors for Thompson Sampling in Multiparameter Bandit Models [56.31310344616837]
  Thompson sampling (TS) has been known for its outstanding empirical performance supported by theoretical guarantees across various reward models. This study explores the impact of selecting noninformative priors, offering insights into the performance of TS when dealing with new models that lack theoretical understanding.
  arXiv  Detail & Related papers  (2023-02-28T08:42:42Z)
• Differentially private Riemannian optimization [40.23168342389821]
  We introduce a framework of differentially private. Ritual risk problem where the. parameter is constrained to a. Rockefellerian manifold. We show the efficacy of the proposed framework in several applications.
  arXiv  Detail & Related papers  (2022-05-19T12:04:15Z)
• Square Root Marginalization for Sliding-Window Bundle Adjustment [53.34552451834707]
  We propose a novel square root sliding-window bundle adjustment suitable for real-time odometry applications. We show that the proposed square root marginalization is algebraically equivalent to the conventional use of Schur complement (SC) on the Hessian. Our evaluation of visual and visual-inertial odometry on real-world datasets demonstrates that the proposed estimator is 36% faster than the baseline.
  arXiv  Detail & Related papers  (2021-09-05T23:22:38Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Dimension-robust Function Space MCMC With Neural Network Priors [0.0]
  This paper introduces a new prior on functions spaces which scales more favourably in the dimension of the function's domain. We show that our resulting posterior of the unknown function is amenable to sampling using Hilbert space Markov chain Monte Carlo methods. We show that our priors are competitive and have distinct advantages over other function space priors.
  arXiv  Detail & Related papers  (2020-12-20T14:52:57Z)
• Dynamic sparsity on dynamic regression models [0.0]
  We consider variable selection and shrinkage for the Gaussian dynamic linear regression within a Bayesian framework. We propose a novel method that allows for time-varying sparsity, based on an extension of spike-and-slab priors for dynamic models.
  arXiv  Detail & Related papers  (2020-09-29T16:26:08Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• tvGP-VAE: Tensor-variate Gaussian Process Prior Variational Autoencoder [0.0]
  tvGP-VAE is able to explicitly model correlation via the use of kernel functions. We show that the choice of which correlation structures to explicitly represent in the latent space has a significant impact on model performance.
  arXiv  Detail & Related papers  (2020-06-08T17:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.