PAC-Bayes Beyond Parameter Space: Behavioral Equivalence, Z-Information, and Exact Complexity Decomposition

• URL: http://arxiv.org/abs/2608.11465v1
• Date: Tue, 11 Aug 2026 22:09:17 GMT
• Title: PAC-Bayes Beyond Parameter Space: Behavioral Equivalence, Z-Information, and Exact Complexity Decomposition
• Abstract summary: PAC-Bayes theory provides generalization guarantees by controlling the Kullback--Leibler (KL) divergence over a chosen hypothesis representation.<n>We show that PAC-Bayes KL divergence does not distinguish uncertainty over predictive behavior from variation among behaviorally equivalent realizations.<n>We formalize behavioral equivalence through a measurable behavior map and use measure to decompose probability measures on the configuration space into a distribution over predictive behaviors.
• Score: 14.573126997307916
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: PAC-Bayes theory provides generalization guarantees by controlling the Kullback--Leibler (KL) divergence between posterior and prior distributions over a chosen hypothesis representation. However, predictive risk depends only on the predictive behavior induced by a hypothesis, not on the particular internal realization that implements that behavior. In over-parameterized systems, many distinct configurations induce identical predictive behavior, yet the classical PAC-Bayes KL divergence does not distinguish uncertainty over predictive behavior from variation among behaviorally equivalent realizations. We show that this distinction induces an exact structural decomposition of classical PAC-Bayes complexity. We formalize behavioral equivalence through a measurable behavior map and use measure disintegration to decompose probability measures on the configuration space into a distribution over predictive behaviors and conditional distributions over behavioral fibers. This yields an exact decomposition of the classical PAC-Bayes KL divergence into a behavior-selection term and a realization-level term given by an expected conditional KL within fibers. We define Z-information as the negative of this realization-level contribution: the exact gap between the KL divergence and the complexity of uncertainty over predictive behavior alone. We further show that the behavior-selection term admits an exact variational characterization: it is the minimum KL divergence among all posteriors inducing the same distribution over predictive behaviors, attained by a canonical fiber-symmetrized representative. Finally, we show that symmetry, behavior-preserving directions, fiber geometry, and invariance under fiber-preserving perturbations arise naturally from the same behavior-map structure. Together, these results identify predictive behavior as the natural object of PAC-Bayes complexity.

Related papers

• High-dimensional ridgeless least squares interpolation under spiked covariance structures [3.656751299156262]
  This paper investigates the behavior of the out-of-sample prediction risk of the high-dimensional ridgeless least-squares estimator when the feature dimension $p$ and the sample size $n$ grow proportionally.<n>We show that the signal energy distributed along latent spike directions determines whether overfitting leads to benign, tempered, or catastrophic overfitting.
  arXiv  Detail & Related papers  (2026-08-07T14:40:44Z)
• Smoothness-Based Derandomization of PAC-Bayes Bounds [48.06655334157014]
  We study PAC-Bayes derandomization for smooth loss functions.<n>We show that unbounded passing from the Gibbs predictor to the deterministic predictor at the posterior mean has a precise cost.<n>We specialize the results to linear predictors and smooth neural networks.
  arXiv  Detail & Related papers  (2026-06-17T14:17:44Z)
• On the Structural Non-Preservation of Epistemic Behaviour under Policy Transformation [51.56484100374058]
  We formalise such information-conditioned interaction patterns as behavioural dependency.<n>This induces a probe-relative notion of $$-behavioural equivalence and a within-policy behavioural distance.<n>Results identify structural conditions under which probe-conditioned behavioural separation is not preserved under common policy transformations.
  arXiv  Detail & Related papers  (2026-02-24T22:55:21Z)
• Credal Concept Bottleneck Models: Structural Separation of Epistemic and Aleatoric Uncertainty [38.040646841317965]
  We propose a credal-set formulation in which uncertainty is represented as a set of predictive distributions.<n>We instantiate this idea in a Variational Credal Concept Bottleneck Model with two disjoint uncertainty heads trained by disjoint objectives.
  arXiv  Detail & Related papers  (2026-02-11T10:54:57Z)
• Counterfactual Realizability [52.85109506684737]
  We introduce a formal definition of realizability, the ability to draw samples from a distribution, and then develop a complete algorithm to determine whether an arbitrary counterfactual distribution is realizable.<n>We illustrate the implications of this new framework for counterfactual data collection using motivating examples from causal fairness and causal reinforcement learning.
  arXiv  Detail & Related papers  (2025-03-14T20:54:27Z)
• Causal vs. Anticausal merging of predictors [57.26526031579287]
  We study the differences arising from merging predictors in the causal and anticausal directions using the same data.<n>We use Causal Maximum Entropy (CMAXENT) as inductive bias to merge the predictors.
  arXiv  Detail & Related papers  (2025-01-14T20:38:15Z)
• Learning Invariant Representations under General Interventions on the Response [2.725698729450241]
  We focus on linear structural causal models (SCMs) and introduce invariant matching property (IMP) We analyze the generalization errors of our method under both the discrete and continuous environment settings.
  arXiv  Detail & Related papers  (2022-08-22T03:09:17Z)
• Discovering Latent Causal Variables via Mechanism Sparsity: A New Principle for Nonlinear ICA [81.4991350761909]
  Independent component analysis (ICA) refers to an ensemble of methods which formalize this goal and provide estimation procedure for practical application. We show that the latent variables can be recovered up to a permutation if one regularizes the latent mechanisms to be sparse.
  arXiv  Detail & Related papers  (2021-07-21T14:22:14Z)
• Which Invariance Should We Transfer? A Causal Minimax Learning Approach [18.71316951734806]
  We present a comprehensive minimax analysis from a causal perspective. We propose an efficient algorithm to search for the subset with minimal worst-case risk. The effectiveness and efficiency of our methods are demonstrated on synthetic data and the diagnosis of Alzheimer's disease.
  arXiv  Detail & Related papers  (2021-07-05T09:07:29Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.