Squared Neural Families: A New Class of Tractable Density Models

• URL: http://arxiv.org/abs/2305.13552v2
• Date: Wed, 25 Oct 2023 22:56:52 GMT
• Title: Squared Neural Families: A New Class of Tractable Density Models
• Authors: Russell Tsuchida and Cheng Soon Ong and Dino Sejdinovic
• Abstract summary: We develop and investigate a new class of probability distributions, which we call a Squared Neural Family (SNEFY) We show that SNEFYs admit closed form normalising constants in many cases of interest, thereby resulting in flexible yet fully tractable density models. Their utility is illustrated on a variety of density estimation, conditional density estimation, and density estimation with missing data tasks.
• Score: 23.337256081314518
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Flexible models for probability distributions are an essential ingredient in many machine learning tasks. We develop and investigate a new class of probability distributions, which we call a Squared Neural Family (SNEFY), formed by squaring the 2-norm of a neural network and normalising it with respect to a base measure. Following the reasoning similar to the well established connections between infinitely wide neural networks and Gaussian processes, we show that SNEFYs admit closed form normalising constants in many cases of interest, thereby resulting in flexible yet fully tractable density models. SNEFYs strictly generalise classical exponential families, are closed under conditioning, and have tractable marginal distributions. Their utility is illustrated on a variety of density estimation, conditional density estimation, and density estimation with missing data tasks.

Related papers

• Score-based generative models break the curse of dimensionality in learning a family of sub-Gaussian probability distributions [5.801621787540268]
  We introduce a notion of complexity for probability distributions in terms of their relative density with respect to the standard Gaussian measure. We prove that if the log-relative density can be locally approximated by a neural network whose parameters can be suitably bounded, then the distribution generated by empirical score matching approximates the target distribution. An essential ingredient of our proof is to derive a dimension-free deep neural network approximation rate for the true score function associated with the forward process.
  arXiv  Detail & Related papers  (2024-02-12T22:02:23Z)
• Numerically assisted determination of local models in network scenarios [55.2480439325792]
  We develop a numerical tool for finding explicit local models that reproduce a given statistical behaviour. We provide conjectures for the critical visibilities of the Greenberger-Horne-Zeilinger (GHZ) and W distributions. The developed codes and documentation are publicly available at281.com/mariofilho/localmodels.
  arXiv  Detail & Related papers  (2023-03-17T13:24:04Z)
• Copula Density Neural Estimation [4.86067125387358]
  We exploit the concept of copula to build an estimate of the probability density function associated to any observed data. Results show that the novel learning approach is capable of modeling complex distributions and it can be applied for mutual information estimation and data generation.
  arXiv  Detail & Related papers  (2022-11-25T10:53:27Z)
• Robust Estimation for Nonparametric Families via Generative Adversarial Networks [92.64483100338724]
  We provide a framework for designing Generative Adversarial Networks (GANs) to solve high dimensional robust statistics problems. Our work extend these to robust mean estimation, second moment estimation, and robust linear regression. In terms of techniques, our proposed GAN losses can be viewed as a smoothed and generalized Kolmogorov-Smirnov distance.
  arXiv  Detail & Related papers  (2022-02-02T20:11:33Z)
• HyperSPNs: Compact and Expressive Probabilistic Circuits [89.897635970366]
  HyperSPNs is a new paradigm of generating the mixture weights of large PCs using a small-scale neural network. We show the merits of our regularization strategy on two state-of-the-art PC families introduced in recent literature.
  arXiv  Detail & Related papers  (2021-12-02T01:24:43Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• Marginalizable Density Models [14.50261153230204]
  We present a novel deep network architecture which provides closed form expressions for the probabilities, marginals and conditionals of any subset of the variables. The model also allows for parallelized sampling with only a logarithmic dependence of the time complexity on the number of variables.
  arXiv  Detail & Related papers  (2021-06-08T23:54:48Z)
• Deconvolutional Density Network: Free-Form Conditional Density Estimation [6.805003206706124]
  A neural network can be used to compute the output distribution explicitly. We show the benefits of modeling free-form distributions using deconvolution. We compare our method to a number of other density-estimation approaches.
  arXiv  Detail & Related papers  (2021-05-29T20:09:25Z)
• Investigating maximum likelihood based training of infinite mixtures for uncertainty quantification [16.30200782698554]
  We investigate the effect of training an infinite mixture distribution with the maximum likelihood method instead of variational inference. We find that the proposed objective leads to adversarial networks with an increased predictive variance.
  arXiv  Detail & Related papers  (2020-08-07T14:55:53Z)
• GANs with Conditional Independence Graphs: On Subadditivity of Probability Divergences [70.30467057209405]
  Generative Adversarial Networks (GANs) are modern methods to learn the underlying distribution of a data set. GANs are designed in a model-free fashion where no additional information about the underlying distribution is available. We propose a principled design of a model-based GAN that uses a set of simple discriminators on the neighborhoods of the Bayes-net/MRF.
  arXiv  Detail & Related papers  (2020-03-02T04:31:22Z)
• Learning Likelihoods with Conditional Normalizing Flows [54.60456010771409]
  Conditional normalizing flows (CNFs) are efficient in sampling and inference. We present a study of CNFs where the base density to output space mapping is conditioned on an input x, to model conditional densities p(y|x)
  arXiv  Detail & Related papers  (2019-11-29T19:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.