On the Universality of Coupling-based Normalizing Flows

• URL: http://arxiv.org/abs/2402.06578v2
• Date: Wed, 5 Jun 2024 17:52:14 GMT
• Title: On the Universality of Coupling-based Normalizing Flows
• Authors: Felix Draxler, Stefan Wahl, Christoph Schnörr, Ullrich Köthe,
• Abstract summary: We propose a distributional theorem for well-conditioned coupling-based normalizing flows such as RealNVP. We show that volume-preserving normalizing flows are not universal, what distribution they learn instead, and how to fix their expressivity.
• Score: 10.479969050570684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a novel theoretical framework for understanding the expressive power of normalizing flows. Despite their prevalence in scientific applications, a comprehensive understanding of flows remains elusive due to their restricted architectures. Existing theorems fall short as they require the use of arbitrarily ill-conditioned neural networks, limiting practical applicability. We propose a distributional universality theorem for well-conditioned coupling-based normalizing flows such as RealNVP. In addition, we show that volume-preserving normalizing flows are not universal, what distribution they learn instead, and how to fix their expressivity. Our results support the general wisdom that affine and related couplings are expressive and in general outperform volume-preserving flows, bridging a gap between empirical results and theoretical understanding.

Related papers

• Interpolation, Approximation and Controllability of Deep Neural Networks [18.311411538309425]
  We consider two properties that arise from supervised learning, namely universal and universal approximation. We give a characterisation of universal equivalence, showing that it holds for essentially any architecture with non-linearity.
  arXiv  Detail & Related papers  (2023-09-12T07:29:47Z)
• Unifying different notions of quantum incompatibility into a strict hierarchy of resource theories of communication [60.18814584837969]
  We introduce the notion of q-compatibility, which unifies different notions of POVMs, channels, and instruments incompatibility. We are able to pinpoint exactly what each notion of incompatibility consists of, in terms of information-theoretic resources.
  arXiv  Detail & Related papers  (2022-11-16T21:33:31Z)
• Whitening Convergence Rate of Coupling-based Normalizing Flows [1.1279808969568252]
  Existing work shows that such flows weakly converge to arbitrary data distributions. We prove that all coupling-based normalizing flows perform whitening of the data distribution. We derive corresponding convergence bounds that show a linear convergence rate in the depth of the flow.
  arXiv  Detail & Related papers  (2022-10-25T14:10:34Z)
• Universal Approximation for Log-concave Distributions using Well-conditioned Normalizing Flows [20.022920482589324]
  We show that any log-concave distribution can be approximated using well-conditioned affine-coupling flows. Our results also inform the practice of training affine couplings.
  arXiv  Detail & Related papers  (2021-07-07T00:13:50Z)
• Measuring Generalization with Optimal Transport [111.29415509046886]
  We develop margin-based generalization bounds, where the margins are normalized with optimal transport costs. Our bounds robustly predict the generalization error, given training data and network parameters, on large scale datasets.
  arXiv  Detail & Related papers  (2021-06-07T03:04:59Z)
• Universal Approximation of Residual Flows in Maximum Mean Discrepancy [24.493721984271566]
  We study residual flows, a class of normalizing flows composed of Lipschitz residual blocks. We prove residual flows are universal approximators in maximum mean discrepancy.
  arXiv  Detail & Related papers  (2021-03-10T00:16:33Z)
• In Search of Robust Measures of Generalization [79.75709926309703]
  We develop bounds on generalization error, optimization error, and excess risk. When evaluated empirically, most of these bounds are numerically vacuous. We argue that generalization measures should instead be evaluated within the framework of distributional robustness.
  arXiv  Detail & Related papers  (2020-10-22T17:54:25Z)
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators [72.62940905965267]
  Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. Are CF-INNs universal approximators for invertible functions? We prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes.
  arXiv  Detail & Related papers  (2020-06-20T02:07:37Z)
• Generalised Lipschitz Regularisation Equals Distributional Robustness [47.44261811369141]
  We give a very general equality result regarding the relationship between distributional robustness and regularisation. We show a new result explicating the connection between adversarial learning and distributional robustness.
  arXiv  Detail & Related papers  (2020-02-11T04:19:43Z)
• Understanding Generalization in Deep Learning via Tensor Methods [53.808840694241]
  We advance the understanding of the relations between the network's architecture and its generalizability from the compression perspective. We propose a series of intuitive, data-dependent and easily-measurable properties that tightly characterize the compressibility and generalizability of neural networks.
  arXiv  Detail & Related papers  (2020-01-14T22:26:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.