On the capacity of deep generative networks for approximating distributions

• URL: http://arxiv.org/abs/2101.12353v1
• Date: Fri, 29 Jan 2021 01:45:02 GMT
• Title: On the capacity of deep generative networks for approximating distributions
• Authors: Yunfei Yang, Zhen Li, Yang Wang
• Abstract summary: We prove that neural networks can transform a one-dimensional source distribution to a distribution arbitrarily close to a high-dimensional target distribution in Wasserstein distances. It is shown that the approximation error grows at most linearly on the ambient dimension. $f$-divergences are less adequate than Waserstein distances as metrics of distributions for generating samples.
• Score: 8.798333793391544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the efficacy and efficiency of deep generative networks for approximating probability distributions. We prove that neural networks can transform a one-dimensional source distribution to a distribution that is arbitrarily close to a high-dimensional target distribution in Wasserstein distances. Upper bounds of the approximation error are obtained in terms of neural networks' width and depth. It is shown that the approximation error grows at most linearly on the ambient dimension and that the approximation order only depends on the intrinsic dimension of the target distribution. On the contrary, when $f$-divergences are used as metrics of distributions, the approximation property is different. We prove that in order to approximate the target distribution in $f$-divergences, the dimension of the source distribution cannot be smaller than the intrinsic dimension of the target distribution. Therefore, $f$-divergences are less adequate than Waserstein distances as metrics of distributions for generating samples.

Related papers

• Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers [49.97755400231656]
  We present the first performance guarantee with explicit dimensional general score-mismatched diffusion samplers. We show that score mismatches result in an distributional bias between the target and sampling distributions, proportional to the accumulated mismatch between the target and training distributions. This result can be directly applied to zero-shot conditional samplers for any conditional model, irrespective of measurement noise.
  arXiv  Detail & Related papers  (2024-10-17T16:42:12Z)
• Uncertainty Quantification via Stable Distribution Propagation [60.065272548502]
  We propose a new approach for propagating stable probability distributions through neural networks. Our method is based on local linearization, which we show to be an optimal approximation in terms of total variation distance for the ReLU non-linearity.
  arXiv  Detail & Related papers  (2024-02-13T09:40:19Z)
• Diffusion Models are Minimax Optimal Distribution Estimators [49.47503258639454]
  We provide the first rigorous analysis on approximation and generalization abilities of diffusion modeling. We show that when the true density function belongs to the Besov space and the empirical score matching loss is properly minimized, the generated data distribution achieves the nearly minimax optimal estimation rates.
  arXiv  Detail & Related papers  (2023-03-03T11:31:55Z)
• Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data [68.62134204367668]
  This paper studies score approximation, estimation, and distribution recovery of diffusion models, when data are supported on an unknown low-dimensional linear subspace. We show that with a properly chosen neural network architecture, the score function can be both accurately approximated and efficiently estimated. The generated distribution based on the estimated score function captures the data geometric structures and converges to a close vicinity of the data distribution.
  arXiv  Detail & Related papers  (2023-02-14T17:02:35Z)
• Learning Distributions by Generative Adversarial Networks: Approximation and Generalization [0.6768558752130311]
  We study how well generative adversarial networks learn from finite samples by analyzing the convergence rates of these models. Our analysis is based on a new inequality oracle that decomposes the estimation error of GAN into the discriminator and generator approximation errors. For generator approximation error, we show that neural network can approximately transform a low-dimensional source distribution to a high-dimensional target distribution.
  arXiv  Detail & Related papers  (2022-05-25T09:26:17Z)
• Robust Estimation for Nonparametric Families via Generative Adversarial Networks [92.64483100338724]
  We provide a framework for designing Generative Adversarial Networks (GANs) to solve high dimensional robust statistics problems. Our work extend these to robust mean estimation, second moment estimation, and robust linear regression. In terms of techniques, our proposed GAN losses can be viewed as a smoothed and generalized Kolmogorov-Smirnov distance.
  arXiv  Detail & Related papers  (2022-02-02T20:11:33Z)
• High-Dimensional Distribution Generation Through Deep Neural Networks [2.141079906482723]
  We show that every $d$-dimensional probability distribution of bounded support can be generated through deep ReLU networks. We find that, for histogram target distributions, the number of bits needed to encode the corresponding generative network equals the fundamental limit for encoding probability distributions.
  arXiv  Detail & Related papers  (2021-07-26T20:35:52Z)
• An error analysis of generative adversarial networks for learning distributions [11.842861158282265]
  generative adversarial networks (GANs) learn probability distributions from finite samples. GANs are able to adaptively learn data distributions with low-dimensional structure or have H"older densities. Our analysis is based on a new oracle inequality decomposing the estimation error into generator and discriminator approximation error and statistical error.
  arXiv  Detail & Related papers  (2021-05-27T08:55:19Z)
• Linear Optimal Transport Embedding: Provable Wasserstein classification for certain rigid transformations and perturbations [79.23797234241471]
  Discriminating between distributions is an important problem in a number of scientific fields. The Linear Optimal Transportation (LOT) embeds the space of distributions into an $L2$-space. We demonstrate the benefits of LOT on a number of distribution classification problems.
  arXiv  Detail & Related papers  (2020-08-20T19:09:33Z)
• Constructive Universal High-Dimensional Distribution Generation through Deep ReLU Networks [3.4591414173342647]
  We present an explicit deep neural network construction that transforms uniformly distributed one-dimensional noise into an arbitrarily close approximation of any two-dimensional Lipschitz-continuous target distribution. We elicit the importance of depth - in our neural network construction - in driving the Wasserstein distance between the target distribution and the approximation realized by the network to zero.
  arXiv  Detail & Related papers  (2020-06-30T10:36:15Z)
• Fast Predictive Uncertainty for Classification with Bayesian Deep Networks [25.821401066200504]
  In Bayesian Deep Learning, distributions over the output of classification neural networks are approximated by first constructing a Gaussian distribution over the weights, then sampling from it to receive a distribution over the softmax outputs. We construct a Dirichlet approximation of this softmax output distribution, which yields an analytic map between Gaussian distributions in logit space and Dirichlet distributions in the output space. We demonstrate that the resulting Dirichlet distribution has multiple advantages, in particular, more efficient of the uncertainty estimate and scaling to large datasets and networks like ImageNet and DenseNet.
  arXiv  Detail & Related papers  (2020-03-02T22:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.