HyperSPNs: Compact and Expressive Probabilistic Circuits

• URL: http://arxiv.org/abs/2112.00914v1
• Date: Thu, 2 Dec 2021 01:24:43 GMT
• Title: HyperSPNs: Compact and Expressive Probabilistic Circuits
• Authors: Andy Shih and Dorsa Sadigh and Stefano Ermon
• Abstract summary: HyperSPNs is a new paradigm of generating the mixture weights of large PCs using a small-scale neural network. We show the merits of our regularization strategy on two state-of-the-art PC families introduced in recent literature.
• Score: 89.897635970366
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Probabilistic circuits (PCs) are a family of generative models which allows for the computation of exact likelihoods and marginals of its probability distributions. PCs are both expressive and tractable, and serve as popular choices for discrete density estimation tasks. However, large PCs are susceptible to overfitting, and only a few regularization strategies (e.g., dropout, weight-decay) have been explored. We propose HyperSPNs: a new paradigm of generating the mixture weights of large PCs using a small-scale neural network. Our framework can be viewed as a soft weight-sharing strategy, which combines the greater expressiveness of large models with the better generalization and memory-footprint properties of small models. We show the merits of our regularization strategy on two state-of-the-art PC families introduced in recent literature -- RAT-SPNs and EiNETs -- and demonstrate generalization improvements in both models on a suite of density estimation benchmarks in both discrete and continuous domains.

Related papers

• Sum of Squares Circuits [8.323409122604893]
  Probabilistic circuits (PCs) offer a framework where this tractability-vs-expressiveness trade-off can be analyzed theoretically. We show that squared PCs encoding subtractive mixtures via negative parameters can be exponentially more expressive than monotonic PCs. We formalize a novel class of PCs -- sum of squares PCs -- that can be exponentially more expressive than both squared and monotonic PCs.
  arXiv  Detail & Related papers  (2024-08-21T17:08:05Z)
• Probabilistic Neural Circuits [4.724177741282789]
  Probabilistic neural circuits (PNCs) strike a balance between PCs and neural nets in terms of tractability and expressive power. We show that PNCs can be interpreted as deep mixtures of Bayesian networks.
  arXiv  Detail & Related papers  (2024-03-10T15:25:49Z)
• PAC Reinforcement Learning for Predictive State Representations [60.00237613646686]
  We study online Reinforcement Learning (RL) in partially observable dynamical systems. We focus on the Predictive State Representations (PSRs) model, which is an expressive model that captures other well-known models. We develop a novel model-based algorithm for PSRs that can learn a near optimal policy in sample complexity scalingly.
  arXiv  Detail & Related papers  (2022-07-12T17:57:17Z)
• A new perspective on probabilistic image modeling [92.89846887298852]
  We present a new probabilistic approach for image modeling capable of density estimation, sampling and tractable inference. DCGMMs can be trained end-to-end by SGD from random initial conditions, much like CNNs. We show that DCGMMs compare favorably to several recent PC and SPN models in terms of inference, classification and sampling.
  arXiv  Detail & Related papers  (2022-03-21T14:53:57Z)
• Scaling Structured Inference with Randomization [64.18063627155128]
  We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states. Our method is widely applicable to classical DP-based inference. It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
  arXiv  Detail & Related papers  (2021-12-07T11:26:41Z)
• Tractable Regularization of Probabilistic Circuits [31.841838579553034]
  Probabilistic Circuits (PCs) are a promising avenue for probabilistic modeling. We propose two intuitive techniques, data softening and entropy regularization, that take advantage of PCs' tractability. We show that both methods consistently improve the generalization performance of a wide variety of PCs.
  arXiv  Detail & Related papers  (2021-06-04T05:11:13Z)
• Probabilistic Generating Circuits [50.98473654244851]
  We propose probabilistic generating circuits (PGCs) for their efficient representation. PGCs are not just a theoretical framework that unifies vastly different existing models, but also show huge potential in modeling realistic data. We exhibit a simple class of PGCs that are not trivially subsumed by simple combinations of PCs and DPPs, and obtain competitive performance on a suite of density estimation benchmarks.
  arXiv  Detail & Related papers  (2021-02-19T07:06:53Z)
• Einsum Networks: Fast and Scalable Learning of Tractable Probabilistic Circuits [99.59941892183454]
  We propose Einsum Networks (EiNets), a novel implementation design for PCs. At their core, EiNets combine a large number of arithmetic operations in a single monolithic einsum-operation. We show that the implementation of Expectation-Maximization (EM) can be simplified for PCs, by leveraging automatic differentiation.
  arXiv  Detail & Related papers  (2020-04-13T23:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.