PAC-Bayes Analysis Beyond the Usual Bounds

• URL: http://arxiv.org/abs/2006.13057v3
• Date: Mon, 28 Dec 2020 18:59:11 GMT
• Title: PAC-Bayes Analysis Beyond the Usual Bounds
• Authors: Omar Rivasplata, Ilja Kuzborskij, Csaba Szepesvari, John Shawe-Taylor
• Abstract summary: We focus on a learning model where the learner observes a finite set of training examples. The learned data-dependent distribution is then used to make randomized predictions.
• Score: 16.76187007910588
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We focus on a stochastic learning model where the learner observes a finite set of training examples and the output of the learning process is a data-dependent distribution over a space of hypotheses. The learned data-dependent distribution is then used to make randomized predictions, and the high-level theme addressed here is guaranteeing the quality of predictions on examples that were not seen during training, i.e. generalization. In this setting the unknown quantity of interest is the expected risk of the data-dependent randomized predictor, for which upper bounds can be derived via a PAC-Bayes analysis, leading to PAC-Bayes bounds. Specifically, we present a basic PAC-Bayes inequality for stochastic kernels, from which one may derive extensions of various known PAC-Bayes bounds as well as novel bounds. We clarify the role of the requirements of fixed 'data-free' priors, bounded losses, and i.i.d. data. We highlight that those requirements were used to upper-bound an exponential moment term, while the basic PAC-Bayes theorem remains valid without those restrictions. We present three bounds that illustrate the use of data-dependent priors, including one for the unbounded square loss.

Related papers

• Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
  We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We further extend our analysis to the case where the test point has non-trivial correlations with the training set, setting often encountered in time series forecasting. We validate our theory across a variety of high dimensional data.
  arXiv  Detail & Related papers  (2024-08-08T17:27:29Z)
• A finite-sample generalization bound for stable LPV systems [0.0]
  We derive a PAC bound for stable continuous-time linear parameter-varying (LPV) systems. Our bound depends on the H2 norm of the chosen class of the LPV systems, but does not depend on the time interval for which the signals are considered.
  arXiv  Detail & Related papers  (2024-05-16T12:42:36Z)
• Uniform Generalization Bounds on Data-Dependent Hypothesis Sets via PAC-Bayesian Theory on Random Sets [25.250314934981233]
  We first apply the PAC-Bayesian framework on random sets' in a rigorous way, where the training algorithm is assumed to output a data-dependent hypothesis set. This approach allows us to prove data-dependent bounds, which can be applicable in numerous contexts.
  arXiv  Detail & Related papers  (2024-04-26T14:28:18Z)
• Score Matching-based Pseudolikelihood Estimation of Neural Marked Spatio-Temporal Point Process with Uncertainty Quantification [59.81904428056924]
  We introduce SMASH: a Score MAtching estimator for learning markedPs with uncertainty quantification. Specifically, our framework adopts a normalization-free objective by estimating the pseudolikelihood of markedPs through score-matching. The superior performance of our proposed framework is demonstrated through extensive experiments in both event prediction and uncertainty quantification.
  arXiv  Detail & Related papers  (2023-10-25T02:37:51Z)
• Adversarial Resilience in Sequential Prediction via Abstention [46.80218090768711]
  We study the problem of sequential prediction in the setting with an adversary that is allowed to inject clean-label adversarial examples. We propose a new model of sequential prediction that sits between the purely and fully adversarial settings.
  arXiv  Detail & Related papers  (2023-06-22T17:44:22Z)
• Robust PAC$^m$: Training Ensemble Models Under Misspecification and Outliers [46.38465729190199]
  PAC-Bayes theory demonstrates that the free energy criterion minimized by Bayesian learning is a bound on the generalization error for Gibbs predictors. This work presents a novel robust free energy criterion that combines the generalized score function with PAC$m$ ensemble bounds.
  arXiv  Detail & Related papers  (2022-03-03T17:11:07Z)
• Self-Certifying Classification by Linearized Deep Assignment [65.0100925582087]
  We propose a novel class of deep predictors for classifying metric data on graphs within PAC-Bayes risk certification paradigm. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables learning posterior distributions on the hypothesis space.
  arXiv  Detail & Related papers  (2022-01-26T19:59:14Z)
• User-friendly introduction to PAC-Bayes bounds [0.6599344783327052]
  In statistical learning theory there is a set of tools designed to understand the generalization ability of procedures: PAC-Bayesian or PAC-Bayes bounds. Very recently, PAC-Bayes bounds received a considerable attention: for example there was workshop on PAC-Bayesian trends and insights", organized by B. Guedj, F. Bach and P. Germain.
  arXiv  Detail & Related papers  (2021-10-21T15:50:05Z)
• Distribution-free uncertainty quantification for classification under label shift [105.27463615756733]
  We focus on uncertainty quantification (UQ) for classification problems via two avenues. We first argue that label shift hurts UQ, by showing degradation in coverage and calibration. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
  arXiv  Detail & Related papers  (2021-03-04T20:51:03Z)
• On the role of data in PAC-Bayes bounds [24.53731903804468]
  PAC-Bayes bounds are often the Kullback--Leibler divergence between the posterior and prior. We show that the bound based on the prior can be suboptimal. We show that using data can mean the difference between vacuous and nonvacuous bounds.
  arXiv  Detail & Related papers  (2020-06-19T02:03:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.