Continuous Mixtures of Tractable Probabilistic Models
- URL: http://arxiv.org/abs/2209.10584v3
- Date: Fri, 24 Mar 2023 11:58:07 GMT
- Title: Continuous Mixtures of Tractable Probabilistic Models
- Authors: Alvaro H.C. Correia, Gennaro Gala, Erik Quaeghebeur, Cassio de Campos,
Robert Peharz
- Abstract summary: Probabilistic models based on continuous latent spaces, such as variational autoencoders, can be understood as uncountable mixture models.
Probabilistic circuits (PCs) can be understood as hierarchical discrete mixture models.
In this paper, we investigate a hybrid approach, namely continuous mixtures of tractable models with a small latent dimension.
- Score: 10.667104977730304
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Probabilistic models based on continuous latent spaces, such as variational
autoencoders, can be understood as uncountable mixture models where components
depend continuously on the latent code. They have proven to be expressive tools
for generative and probabilistic modelling, but are at odds with tractable
probabilistic inference, that is, computing marginals and conditionals of the
represented probability distribution. Meanwhile, tractable probabilistic models
such as probabilistic circuits (PCs) can be understood as hierarchical discrete
mixture models, and thus are capable of performing exact inference efficiently
but often show subpar performance in comparison to continuous latent-space
models. In this paper, we investigate a hybrid approach, namely continuous
mixtures of tractable models with a small latent dimension. While these models
are analytically intractable, they are well amenable to numerical integration
schemes based on a finite set of integration points. With a large enough number
of integration points the approximation becomes de-facto exact. Moreover, for a
finite set of integration points, the integration method effectively compiles
the continuous mixture into a standard PC. In experiments, we show that this
simple scheme proves remarkably effective, as PCs learnt this way set new state
of the art for tractable models on many standard density estimation benchmarks.
Related papers
- Fusion of Gaussian Processes Predictions with Monte Carlo Sampling [61.31380086717422]
In science and engineering, we often work with models designed for accurate prediction of variables of interest.
Recognizing that these models are approximations of reality, it becomes desirable to apply multiple models to the same data and integrate their outcomes.
arXiv Detail & Related papers (2024-03-03T04:21:21Z) - Simplifying Complex Observation Models in Continuous POMDP Planning with
Probabilistic Guarantees and Practice [9.444784653236157]
We deal with the question of what would be the implication of using simplified observation models for planning.
Our main contribution is a novel probabilistic bound based on a statistical total variation distance of the simplified model.
Our calculations can be separated into offline and online parts, and we arrive at formal guarantees without having to access the costly model at all during planning.
arXiv Detail & Related papers (2023-11-13T20:55:02Z) - Scalable Dynamic Mixture Model with Full Covariance for Probabilistic
Traffic Forecasting [16.04029885574568]
We propose a dynamic mixture of zero-mean Gaussian distributions for the time-varying error process.
The proposed method can be seamlessly integrated into existing deep-learning frameworks with only a few additional parameters to be learned.
We evaluate the proposed method on a traffic speed forecasting task and find that our method not only improves model horizons but also provides interpretabletemporal correlation structures.
arXiv Detail & Related papers (2022-12-10T22:50:00Z) - Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain.
We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions.
We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
arXiv Detail & Related papers (2022-11-30T05:33:29Z) - Statistical Efficiency of Score Matching: The View from Isoperimetry [96.65637602827942]
We show a tight connection between statistical efficiency of score matching and the isoperimetric properties of the distribution being estimated.
We formalize these results both in the sample regime and in the finite regime.
arXiv Detail & Related papers (2022-10-03T06:09:01Z) - PSD Representations for Effective Probability Models [117.35298398434628]
We show that a recently proposed class of positive semi-definite (PSD) models for non-negative functions is particularly suited to this end.
We characterize both approximation and generalization capabilities of PSD models, showing that they enjoy strong theoretical guarantees.
Our results open the way to applications of PSD models to density estimation, decision theory and inference.
arXiv Detail & Related papers (2021-06-30T15:13:39Z) - Marginalizable Density Models [14.50261153230204]
We present a novel deep network architecture which provides closed form expressions for the probabilities, marginals and conditionals of any subset of the variables.
The model also allows for parallelized sampling with only a logarithmic dependence of the time complexity on the number of variables.
arXiv Detail & Related papers (2021-06-08T23:54:48Z) - Probabilistic Generating Circuits [50.98473654244851]
We propose probabilistic generating circuits (PGCs) for their efficient representation.
PGCs are not just a theoretical framework that unifies vastly different existing models, but also show huge potential in modeling realistic data.
We exhibit a simple class of PGCs that are not trivially subsumed by simple combinations of PCs and DPPs, and obtain competitive performance on a suite of density estimation benchmarks.
arXiv Detail & Related papers (2021-02-19T07:06:53Z) - Probabilistic Circuits for Variational Inference in Discrete Graphical
Models [101.28528515775842]
Inference in discrete graphical models with variational methods is difficult.
Many sampling-based methods have been proposed for estimating Evidence Lower Bound (ELBO)
We propose a new approach that leverages the tractability of probabilistic circuit models, such as Sum Product Networks (SPN)
We show that selective-SPNs are suitable as an expressive variational distribution, and prove that when the log-density of the target model is aweighted the corresponding ELBO can be computed analytically.
arXiv Detail & Related papers (2020-10-22T05:04:38Z) - Variational Mixture of Normalizing Flows [0.0]
Deep generative models, such as generative adversarial networks autociteGAN, variational autoencoders autocitevaepaper, and their variants, have seen wide adoption for the task of modelling complex data distributions.
Normalizing flows have overcome this limitation by leveraging the change-of-suchs formula for probability density functions.
The present work overcomes this by using normalizing flows as components in a mixture model and devising an end-to-end training procedure for such a model.
arXiv Detail & Related papers (2020-09-01T17:20:08Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.