Approximating Probability Distributions by ReLU Networks

• URL: http://arxiv.org/abs/2101.09973v1
• Date: Mon, 25 Jan 2021 09:31:20 GMT
• Title: Approximating Probability Distributions by ReLU Networks
• Abstract summary: How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds.
• Score: 15.899159309486679
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.

Related papers

• Accuracy estimation of neural networks by extreme value theory [0.0]
  It is not obvious how to quantify the error of the neural network, i.e., the remaining bias between the function and the neural network.<n>We propose the application of extreme value theory to quantify large values of the error.
  arXiv  Detail & Related papers  (2025-11-01T10:52:51Z)
• Generative Conditional Distributions by Neural (Entropic) Optimal Transport [12.152228552335798]
  We introduce a novel neural entropic optimal transport method designed to learn generative models of conditional distributions. Our method relies on the minimax training of two neural networks. Our experiments on real-world datasets show the effectiveness of our algorithm compared to state-of-the-art conditional distribution learning techniques.
  arXiv  Detail & Related papers  (2024-06-04T13:45:35Z)
• A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints [87.08677547257733]
  Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning. We show how to maximize the likelihood of a symbolic constraint w.r.t the neural network's output distribution. We also evaluate our approach on Sudoku and shortest-path prediction cast as autoregressive generation.
  arXiv  Detail & Related papers  (2023-12-06T20:58:07Z)
• Learning Theory of Distribution Regression with Neural Networks [6.961253535504979]
  We establish an approximation theory and a learning theory of distribution regression via a fully connected neural network (FNN) In contrast to the classical regression methods, the input variables of distribution regression are probability measures.
  arXiv  Detail & Related papers  (2023-07-07T09:49:11Z)
• Semantic Strengthening of Neuro-Symbolic Learning [85.6195120593625]
  Neuro-symbolic approaches typically resort to fuzzy approximations of a probabilistic objective. We show how to compute this efficiently for tractable circuits. We test our approach on three tasks: predicting a minimum-cost path in Warcraft, predicting a minimum-cost perfect matching, and solving Sudoku puzzles.
  arXiv  Detail & Related papers  (2023-02-28T00:04:22Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Robust Estimation for Nonparametric Families via Generative Adversarial Networks [92.64483100338724]
  We provide a framework for designing Generative Adversarial Networks (GANs) to solve high dimensional robust statistics problems. Our work extend these to robust mean estimation, second moment estimation, and robust linear regression. In terms of techniques, our proposed GAN losses can be viewed as a smoothed and generalized Kolmogorov-Smirnov distance.
  arXiv  Detail & Related papers  (2022-02-02T20:11:33Z)
• Approximation bounds for norm constrained neural networks with applications to regression and GANs [9.645327615996914]
  We prove upper and lower bounds on the approximation error of ReLU neural networks with norm constraint on the weights. We apply these approximation bounds to analyze the convergences of regression using norm constrained neural networks and distribution estimation by GANs.
  arXiv  Detail & Related papers  (2022-01-24T02:19:05Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• Arithmetic Distribution Neural Network for Background Subtraction [7.09875977818162]
  We propose a new Arithmetic Distribution Neural Network (ADNN) for learning the distributions of temporal pixels during background subtraction. The proposed approach is able to utilize the probability information of the histogram and achieve promising results.
  arXiv  Detail & Related papers  (2021-04-16T22:44:58Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• On the capacity of deep generative networks for approximating distributions [8.798333793391544]
  We prove that neural networks can transform a one-dimensional source distribution to a distribution arbitrarily close to a high-dimensional target distribution in Wasserstein distances. It is shown that the approximation error grows at most linearly on the ambient dimension. $f$-divergences are less adequate than Waserstein distances as metrics of distributions for generating samples.
  arXiv  Detail & Related papers  (2021-01-29T01:45:02Z)
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need [56.25432563818297]
  The lottery ticket hypothesis states that a randomly-d network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network. We prove an even stronger hypothesis, showing that for every bounded distribution and every target network with bounded weights, a sufficiently over- parameterized neural network with random weights contains a subnetwork with roughly the same accuracy as the target network, without any further training.
  arXiv  Detail & Related papers  (2020-02-03T07:23:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.