You say Normalizing Flows I see Bayesian Networks

• URL: http://arxiv.org/abs/2006.00866v2
• Date: Wed, 3 Jun 2020 21:43:28 GMT
• Title: You say Normalizing Flows I see Bayesian Networks
• Authors: Antoine Wehenkel and Gilles Louppe
• Abstract summary: We show that normalizing flows reduce to Bayesian networks with a pre-defined topology and a learnable density at each node. We show that stacking multiple transformations in a normalizing flow relaxes independence assumptions and entangles the model distribution. We prove the non-universality of the affine normalizing flow, regardless of its depth.
• Score: 11.23030807455021
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Normalizing flows have emerged as an important family of deep neural networks for modelling complex probability distributions. In this note, we revisit their coupling and autoregressive transformation layers as probabilistic graphical models and show that they reduce to Bayesian networks with a pre-defined topology and a learnable density at each node. From this new perspective, we provide three results. First, we show that stacking multiple transformations in a normalizing flow relaxes independence assumptions and entangles the model distribution. Second, we show that a fundamental leap of capacity emerges when the depth of affine flows exceeds 3 transformation layers. Third, we prove the non-universality of the affine normalizing flow, regardless of its depth.

Related papers

• Implicit Bias of Mirror Descent for Shallow Neural Networks in Univariate Regression [24.3887959016133]
  We show that mirror flow exhibits lazy training and has the same implicit bias as ordinary gradient flow when the network width tends to infinity. For networks with absolute value activations, we show that mirror flow with scaled potentials induces a rich class of biases, which generally cannot be captured by an RKHS norm.
  arXiv  Detail & Related papers  (2024-10-05T00:43:09Z)
• SE(3) Equivariant Augmented Coupling Flows [16.65770540017618]
  Coupling normalizing flows allow for fast sampling and density evaluation. Standard coupling architecture precludes endowing flows that operate on the Cartesian coordinates of atoms.
  arXiv  Detail & Related papers  (2023-08-20T20:49:15Z)
• Delving into Discrete Normalizing Flows on SO(3) Manifold for Probabilistic Rotation Modeling [30.09829541716024]
  We propose a novel normalizing flow on SO(3) manifold. We show that our rotation normalizing flows significantly outperform the baselines on both unconditional and conditional tasks.
  arXiv  Detail & Related papers  (2023-04-08T06:52:02Z)
• Bayesian Interpolation with Deep Linear Networks [92.1721532941863]
  Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory. We show that linear networks make provably optimal predictions at infinite depth. We also show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth.
  arXiv  Detail & Related papers  (2022-12-29T20:57:46Z)
• GFlowOut: Dropout with Generative Flow Networks [76.59535235717631]
  Monte Carlo Dropout has been widely used as a relatively cheap way for approximate Inference. Recent works show that the dropout mask can be viewed as a latent variable, which can be inferred with variational inference. GFlowOutleverages the recently proposed probabilistic framework of Generative Flow Networks (GFlowNets) to learn the posterior distribution over dropout masks.
  arXiv  Detail & Related papers  (2022-10-24T03:00:01Z)
• ManiFlow: Implicitly Representing Manifolds with Normalizing Flows [145.9820993054072]
  Normalizing Flows (NFs) are flexible explicit generative models that have been shown to accurately model complex real-world data distributions. We propose an optimization objective that recovers the most likely point on the manifold given a sample from the perturbed distribution. Finally, we focus on 3D point clouds for which we utilize the explicit nature of NFs, i.e. surface normals extracted from the gradient of the log-likelihood and the log-likelihood itself.
  arXiv  Detail & Related papers  (2022-08-18T16:07:59Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks [83.58049517083138]
  We consider a two-layer ReLU network trained via gradient descent. We show that SGD is biased towards a simple solution. We also provide empirical evidence that knots at locations distinct from the data points might occur.
  arXiv  Detail & Related papers  (2021-11-03T15:14:20Z)
• Graphical Normalizing Flows [11.23030807455021]
  Normalizing flows model complex probability distributions by combining a base distribution with a series of neural networks. State-of-the-art architectures rely on coupling and autoregressive transformations to lift up invertible functions from scalars to vectors. We propose the graphical normalizing flow, a new invertible transformation with either a prescribed or a learnable graphical structure.
  arXiv  Detail & Related papers  (2020-06-03T21:50:29Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)
• Latent Variable Modelling with Hyperbolic Normalizing Flows [35.1659722563025]
  We introduce a novel normalizing flow over hyperbolic VAEs and Euclidean normalizing flows. Our approach achieves improved performance on density estimation, as well as reconstruction of real-world graph data.
  arXiv  Detail & Related papers  (2020-02-15T07:44:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.