How good is PAC-Bayes at explaining generalisation?

• URL: http://arxiv.org/abs/2503.08231v1
• Date: Tue, 11 Mar 2025 09:51:21 GMT
• Title: How good is PAC-Bayes at explaining generalisation?
• Authors: Antoine Picard-Weibel, Eugenio Clerico, Roman Moscoviz, Benjamin Guedj,
• Abstract summary: We discuss necessary conditions for a PAC-Bayes bound to provide a meaningful generalisation guarantee.<n>Our analysis reveals that the optimal generalisation guarantee depends solely on the distribution of the risk induced by the prior distribution.
• Score: 13.084336891814054
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We discuss necessary conditions for a PAC-Bayes bound to provide a meaningful generalisation guarantee. Our analysis reveals that the optimal generalisation guarantee depends solely on the distribution of the risk induced by the prior distribution. In particular, achieving a target generalisation level is only achievable if the prior places sufficient mass on high-performing predictors. We relate these requirements to the prevalent practice of using data-dependent priors in deep learning PAC-Bayes applications, and discuss the implications for the claim that PAC-Bayes ``explains'' generalisation.

Related papers

• PAC-Bayesian Generalization Guarantees for Fairness on Stochastic and Deterministic Classifiers [8.438034474012044]
  We propose a PAC-Bayesian framework for deriving generalization bounds for fairness.<n>Our framework has two advantages: (i) It applies to a broad class of fairness measures that can be expressed as a risk discrepancy, and (ii) it leads to a self-bounding algorithm.
  arXiv  Detail & Related papers  (2026-02-12T08:49:34Z)
• A Framework for Bounding Deterministic Risk with PAC-Bayes: Applications to Majority Votes [4.664367264604233]
  PAC-Bayes is a popular framework for obtaining generalization guarantees in uncountable hypothesis spaces.<n>We propose a unified framework to extract guarantees holding for a single hypothesis from PAC-Bayesian guarantees.
  arXiv  Detail & Related papers  (2025-10-29T14:38:35Z)
• Optimal Conformal Prediction under Epistemic Uncertainty [61.46247583794497]
  Conformal prediction (CP) is a popular framework for representing uncertainty.<n>We introduce Bernoulli prediction sets (BPS) which produce the smallest prediction sets that ensure conditional coverage.<n>When given first-order predictions, BPS reduces to the well-known adaptive prediction sets (APS)
  arXiv  Detail & Related papers  (2025-05-25T08:32:44Z)
• Misclassification excess risk bounds for PAC-Bayesian classification via convexified loss [0.0]
  PAC-Bayesian bounds are a valuable tool for designing new learning algorithms in machine learning. In this paper we show how to leverage relative bounds in expectation rather than relying on PAC-Bayesian bounds in terms of generalization.
  arXiv  Detail & Related papers  (2024-08-16T11:41:06Z)
• Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regularized Importance Sampling [13.001601860404426]
  We introduce a tractable PAC-Bayesian generalization bound that universally applies to common importance weight regularizations. Our results challenge common understanding, demonstrating the effectiveness of standard IW regularization techniques.
  arXiv  Detail & Related papers  (2024-06-05T16:32:14Z)
• PAC-Bayesian Spectrally-Normalized Bounds for Adversarially Robust Generalization [25.272738030198862]
  Deep neural networks (DNNs) are vulnerable to adversarial attacks. adversarially robust generalization is crucial in establishing defense algorithms against adversarial attacks. This paper focuses on norm-based perturbation complexity, based on a PAC-Bayes approach.
  arXiv  Detail & Related papers  (2023-10-09T22:20:27Z)
• Generalised Likelihood Ratio Testing Adversaries through the Differential Privacy Lens [69.10072367807095]
  Differential Privacy (DP) provides tight upper bounds on the capabilities of optimal adversaries. We relax the assumption of a Neyman--Pearson (NPO) adversary to a Generalized Likelihood Test (GLRT) adversary. This mild relaxation leads to improved privacy guarantees.
  arXiv  Detail & Related papers  (2022-10-24T08:24:10Z)
• Probable Domain Generalization via Quantile Risk Minimization [90.15831047587302]
  Domain generalization seeks predictors which perform well on unseen test distributions. We propose a new probabilistic framework for DG where the goal is to learn predictors that perform well with high probability.
  arXiv  Detail & Related papers  (2022-07-20T14:41:09Z)
• Robust PAC$^m$: Training Ensemble Models Under Misspecification and Outliers [46.38465729190199]
  PAC-Bayes theory demonstrates that the free energy criterion minimized by Bayesian learning is a bound on the generalization error for Gibbs predictors. This work presents a novel robust free energy criterion that combines the generalized score function with PAC$m$ ensemble bounds.
  arXiv  Detail & Related papers  (2022-03-03T17:11:07Z)
• User-friendly introduction to PAC-Bayes bounds [4.612289321094512]
  In statistical learning theory there is a set of tools designed to understand the generalization ability of procedures: PAC-Bayesian or PAC-Bayes bounds.<n>Very recently, PAC-Bayes bounds received a considerable attention: for example there was workshop on PAC-Bayes at NIPS 2017, "(Almost) 50 Shades of Bayesian Learning: PAC-Bayesian trends and insights"<n>An elementary introduction to PAC-Bayes theory is still missing.
  arXiv  Detail & Related papers  (2021-10-21T15:50:05Z)
• Regularization Guarantees Generalization in Bayesian Reinforcement Learning through Algorithmic Stability [48.62272919754204]
  We study generalization in Bayesian RL under the probably approximately correct (PAC) framework. Our main contribution is showing that by adding regularization, the optimal policy becomes stable in an appropriate sense.
  arXiv  Detail & Related papers  (2021-09-24T07:48:34Z)
• Bayesian decision-making under misspecified priors with applications to meta-learning [64.38020203019013]
  Thompson sampling and other sequential decision-making algorithms are popular approaches to tackle explore/exploit trade-offs in contextual bandits. We show that performance degrades gracefully with misspecified priors.
  arXiv  Detail & Related papers  (2021-07-03T23:17:26Z)
• Measuring Generalization with Optimal Transport [111.29415509046886]
  We develop margin-based generalization bounds, where the margins are normalized with optimal transport costs. Our bounds robustly predict the generalization error, given training data and network parameters, on large scale datasets.
  arXiv  Detail & Related papers  (2021-06-07T03:04:59Z)
• PAC$^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime [75.19403612525811]
  This work develops a multi-sample loss which can close the gap by spanning a trade-off between the two risks. Empirical study demonstrates improvement to the predictive distribution.
  arXiv  Detail & Related papers  (2020-10-19T16:08:34Z)
• PAC-Bayes Analysis Beyond the Usual Bounds [16.76187007910588]
  We focus on a learning model where the learner observes a finite set of training examples. The learned data-dependent distribution is then used to make randomized predictions.
  arXiv  Detail & Related papers  (2020-06-23T14:30:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.