Smoothness-Based Derandomization of PAC-Bayes Bounds

• URL: http://arxiv.org/abs/2606.19105v1
• Date: Wed, 17 Jun 2026 14:17:44 GMT
• Title: Smoothness-Based Derandomization of PAC-Bayes Bounds
• Abstract summary: We study PAC-Bayes derandomization for smooth loss functions.<n>We show that unbounded passing from the Gibbs predictor to the deterministic predictor at the posterior mean has a precise cost.<n>We specialize the results to linear predictors and smooth neural networks.
• Score: 48.06655334157014
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study PAC-Bayes derandomization for smooth loss functions. Our goal is to obtain generalization bounds that hold with high probability for deterministic predictors by exploiting smoothness properties of both the loss and the predictor class. We show that passing from the Gibbs predictor to the deterministic predictor at the posterior mean has a precise cost, given by the generalization gap of the Jensen gap class. We control this class through its Rademacher complexity, leading to bounds for deterministic predictors that involve flatness quantities expressed in terms of parameter Jacobians and Hessians of the score map. The framework applies to both bounded and unbounded smooth loss functions, and we specialize the results to linear predictors and smooth neural networks. Finally, the Jacobian and Hessian quantities appearing in the theory motivate a practical regularizer. For BatchNorm networks, we compute this regularizer with respect to effective BatchNorm weights obtained by folding the BatchNorm transformation into the adjacent affine weights. Experiments on CIFAR-10 illustrate the behavior of this regularizer under different batch sizes.

Related papers

• On the Role of Batch Size in Stochastic Conditional Gradient Methods [51.97510367368799]
  We study the role of batch size in conditional gradient methods under a $$-Kurdyka-ojasiewicz ($$-KL) condition.<n>Our study reveals a regime-dependent behavior: increasing the batch size initially improves optimization accuracy but, beyond a critical threshold, the benefits and saturate can degrade performance under a fixed token budget.
  arXiv  Detail & Related papers  (2026-03-22T12:23:41Z)
• Robust Predictive Uncertainty and Double Descent in Contaminated Bayesian Random Features [9.140494844209336]
  We propose a robust Bayesian formulation of random feature (RF) regression that accounts explicitly for prior and likelihood misspecification.<n>We derive explicit and tractable bounds for the resulting lower and upper posterior predictive envelopes.
  arXiv  Detail & Related papers  (2026-02-22T10:50:04Z)
• A principled framework for uncertainty decomposition in TabPFN [3.1500610747796376]
  We present a method for uncertainty decomposition in TabPFN.<n>We derive variance estimators determined by the volatility of predictive updates along the context.<n>For classification, we obtain an entropy-based uncertainty decomposition.
  arXiv  Detail & Related papers  (2026-02-04T14:23:53Z)
• Bayesian stepwise estimation of qubit rotations [0.15650076339927835]
  We investigate stepwise estimation (Se) for measuring the two parameters of a unitary qubit rotation.<n>While analysis predicts a precision advantage for SE over joint estimation (JE) in regimes where the quantum Fisher information matrix is near-singular ("sloppy" models)<n>We demonstrate that this advantage is mitigated within a practical Bayesian framework with limited resources.
  arXiv  Detail & Related papers  (2025-12-04T15:26:06Z)
• Semiparametric conformal prediction [79.6147286161434]
  We construct a conformal prediction set accounting for the joint correlation structure of the vector-valued non-conformity scores.<n>We flexibly estimate the joint cumulative distribution function (CDF) of the scores.<n>Our method yields desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Robust Estimation for Nonparametric Families via Generative Adversarial Networks [92.64483100338724]
  We provide a framework for designing Generative Adversarial Networks (GANs) to solve high dimensional robust statistics problems. Our work extend these to robust mean estimation, second moment estimation, and robust linear regression. In terms of techniques, our proposed GAN losses can be viewed as a smoothed and generalized Kolmogorov-Smirnov distance.
  arXiv  Detail & Related papers  (2022-02-02T20:11:33Z)
• Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise [51.080620762639434]
  We propose a more efficient parameterization of the posterior approximation for sampling-free variational inference. Our approach yields competitive results for standard regression problems and scales well to large-scale image classification tasks.
  arXiv  Detail & Related papers  (2021-03-15T16:16:18Z)
• The Heavy-Tail Phenomenon in SGD [7.366405857677226]
  We show that depending on the structure of the Hessian of the loss at the minimum, the SGD iterates will converge to a emphheavy-tailed stationary distribution. We translate our results into insights about the behavior of SGD in deep learning.
  arXiv  Detail & Related papers  (2020-06-08T16:43:56Z)
• De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors [21.59277717031637]
  We present a family of de-randomized PACes for deterministic non-smooth predictors, e.g., ReLU-nets. We also present empirical results of our bounds over changing set size and in labels.
  arXiv  Detail & Related papers  (2020-02-23T17:54:07Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)
• An Equivalence between Bayesian Priors and Penalties in Variational Inference [8.45602005745865]
  In machine learning, it is common to optimize the parameters of a probabilistic model, modulated by an ad hoc regularization term that penalizes some values of the parameters. We fully characterize the regularizers that can arise according to this procedure, and provide a systematic way to compute the prior corresponding to a given penalty.
  arXiv  Detail & Related papers  (2020-02-01T09:48:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.