Robust Estimation for Nonparametric Families via Generative Adversarial Networks

• URL: http://arxiv.org/abs/2202.01269v1
• Date: Wed, 2 Feb 2022 20:11:33 GMT
• Title: Robust Estimation for Nonparametric Families via Generative Adversarial Networks
• Authors: Banghua Zhu, Jiantao Jiao and Michael I. Jordan
• Abstract summary: We provide a framework for designing Generative Adversarial Networks (GANs) to solve high dimensional robust statistics problems. Our work extend these to robust mean estimation, second moment estimation, and robust linear regression. In terms of techniques, our proposed GAN losses can be viewed as a smoothed and generalized Kolmogorov-Smirnov distance.
• Score: 92.64483100338724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We provide a general framework for designing Generative Adversarial Networks (GANs) to solve high dimensional robust statistics problems, which aim at estimating unknown parameter of the true distribution given adversarially corrupted samples. Prior work focus on the problem of robust mean and covariance estimation when the true distribution lies in the family of Gaussian distributions or elliptical distributions, and analyze depth or scoring rule based GAN losses for the problem. Our work extend these to robust mean estimation, second moment estimation, and robust linear regression when the true distribution only has bounded Orlicz norms, which includes the broad family of sub-Gaussian, sub-Exponential and bounded moment distributions. We also provide a different set of sufficient conditions for the GAN loss to work: we only require its induced distance function to be a cumulative density function of some light-tailed distribution, which is easily satisfied by neural networks with sigmoid activation. In terms of techniques, our proposed GAN losses can be viewed as a smoothed and generalized Kolmogorov-Smirnov distance, which overcomes the computational intractability of the original Kolmogorov-Smirnov distance used in the prior work.

Related papers

• Score-based generative models break the curse of dimensionality in learning a family of sub-Gaussian probability distributions [5.801621787540268]
  We introduce a notion of complexity for probability distributions in terms of their relative density with respect to the standard Gaussian measure. We prove that if the log-relative density can be locally approximated by a neural network whose parameters can be suitably bounded, then the distribution generated by empirical score matching approximates the target distribution. An essential ingredient of our proof is to derive a dimension-free deep neural network approximation rate for the true score function associated with the forward process.
  arXiv  Detail & Related papers  (2024-02-12T22:02:23Z)
• Distributed Markov Chain Monte Carlo Sampling based on the Alternating Direction Method of Multipliers [143.6249073384419]
  In this paper, we propose a distributed sampling scheme based on the alternating direction method of multipliers. We provide both theoretical guarantees of our algorithm's convergence and experimental evidence of its superiority to the state-of-the-art. In simulation, we deploy our algorithm on linear and logistic regression tasks and illustrate its fast convergence compared to existing gradient-based methods.
  arXiv  Detail & Related papers  (2024-01-29T02:08:40Z)
• Learning Distributions by Generative Adversarial Networks: Approximation and Generalization [0.6768558752130311]
  We study how well generative adversarial networks learn from finite samples by analyzing the convergence rates of these models. Our analysis is based on a new inequality oracle that decomposes the estimation error of GAN into the discriminator and generator approximation errors. For generator approximation error, we show that neural network can approximately transform a low-dimensional source distribution to a high-dimensional target distribution.
  arXiv  Detail & Related papers  (2022-05-25T09:26:17Z)
• Optimal 1-Wasserstein Distance for WGANs [2.1174215880331775]
  We provide a thorough analysis of Wasserstein GANs (WGANs) in both the finite sample and regimes. We derive in passing new results on optimal transport theory in the semi-discrete setting.
  arXiv  Detail & Related papers  (2022-01-08T13:04:03Z)
• Decentralized Local Stochastic Extra-Gradient for Variational Inequalities [125.62877849447729]
  We consider distributed variational inequalities (VIs) on domains with the problem data that is heterogeneous (non-IID) and distributed across many devices. We make a very general assumption on the computational network that covers the settings of fully decentralized calculations. We theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone settings.
  arXiv  Detail & Related papers  (2021-06-15T17:45:51Z)
• An error analysis of generative adversarial networks for learning distributions [11.842861158282265]
  generative adversarial networks (GANs) learn probability distributions from finite samples. GANs are able to adaptively learn data distributions with low-dimensional structure or have H"older densities. Our analysis is based on a new oracle inequality decomposing the estimation error into generator and discriminator approximation error and statistical error.
  arXiv  Detail & Related papers  (2021-05-27T08:55:19Z)
• Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise [51.080620762639434]
  We propose a more efficient parameterization of the posterior approximation for sampling-free variational inference. Our approach yields competitive results for standard regression problems and scales well to large-scale image classification tasks.
  arXiv  Detail & Related papers  (2021-03-15T16:16:18Z)
• The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks [4.307812758854161]
  We construct a prior distribution for the parameters of a network that approximates the posited Gaussian process in the output space of the network. This establishes the property that a Bayesian neural network can approximate any Gaussian process whose covariance function is sufficiently regular.
  arXiv  Detail & Related papers  (2020-10-16T16:39:45Z)
• Distribution Approximation and Statistical Estimation Guarantees of Generative Adversarial Networks [82.61546580149427]
  Generative Adversarial Networks (GANs) have achieved a great success in unsupervised learning. This paper provides approximation and statistical guarantees of GANs for the estimation of data distributions with densities in a H"older space.
  arXiv  Detail & Related papers  (2020-02-10T16:47:57Z)
• Distributionally Robust Bayesian Quadrature Optimization [60.383252534861136]
  We study BQO under distributional uncertainty in which the underlying probability distribution is unknown except for a limited set of its i.i.d. samples. A standard BQO approach maximizes the Monte Carlo estimate of the true expected objective given the fixed sample set. We propose a novel posterior sampling based algorithm, namely distributionally robust BQO (DRBQO) for this purpose.
  arXiv  Detail & Related papers  (2020-01-19T12:00:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.