PAC-Bayes Information Bottleneck

• URL: http://arxiv.org/abs/2109.14509v1
• Date: Wed, 29 Sep 2021 15:44:26 GMT
• Title: PAC-Bayes Information Bottleneck
• Abstract summary: Information bottleneck (IB) depicts a trade-off between the accuracy and conciseness of encoded representations. We build a new IB based on the trade-off between the accuracy and complexity of learned weights of NNs. We argue that this new IB represents a more solid connection to the objective of NNs since the information stored in weights bounds their PAC-Bayes generalization capability.
• Score: 44.397128797719425
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Information bottleneck (IB) depicts a trade-off between the accuracy and conciseness of encoded representations. IB has succeeded in explaining the objective and behavior of neural networks (NNs) as well as learning better representations. However, there are still critics of the universality of IB, e.g., phase transition usually fades away, representation compression is not causally related to generalization, and IB is trivial in deterministic cases. In this work, we build a new IB based on the trade-off between the accuracy and complexity of learned weights of NNs. We argue that this new IB represents a more solid connection to the objective of NNs since the information stored in weights bounds their PAC-Bayes generalization capability, hence we name it as PAC-Bayes IB (PIB). On PIB, we can identify the phase transition phenomenon in general cases and solidify the causality between compression and generalization. We then derive a tractable solution of PIB and design a stochastic inference algorithm by Markov chain Monte Carlo sampling. We empirically verify our claims through extensive experiments. We also substantiate the superiority of the proposed algorithm on training better NNs.

Related papers

• A Generalized Information Bottleneck Theory of Deep Learning [10.454976783057086]
  The Information Bottleneck (IB) principle offers a compelling theoretical framework to understand how neural networks (NNs) learn.<n>We present a framework that reformulates the original IB principle through the lens of synergy.
  arXiv  Detail & Related papers  (2025-09-30T14:38:56Z)
• BiDense: Binarization for Dense Prediction [62.70804353158387]
  BiDense is a generalized binary neural network (BNN) designed for efficient and accurate dense prediction tasks. BiDense incorporates two key techniques: the Distribution-adaptive Binarizer (DAB) and the Channel-adaptive Full-precision Bypass (CFB)
  arXiv  Detail & Related papers  (2024-11-15T16:46:04Z)
• Cauchy-Schwarz Divergence Information Bottleneck for Regression [27.768887283866643]
  Information bottleneck (IB) approach is popular to improve generalization, robustness and explainability of deep neural networks. We develop a new way to parameterize the IB with deep neural networks by exploiting favorable properties of the Cauchy-Schwarz divergence. We demonstrate its superior performance on six real-world regression tasks over other popular deep IB approaches.
  arXiv  Detail & Related papers  (2024-04-27T16:13:05Z)
• Graph Out-of-Distribution Generalization via Causal Intervention [69.70137479660113]
  We introduce a conceptually simple yet principled approach for training robust graph neural networks (GNNs) under node-level distribution shifts. Our method resorts to a new learning objective derived from causal inference that coordinates an environment estimator and a mixture-of-expert GNN predictor. Our model can effectively enhance generalization with various types of distribution shifts and yield up to 27.4% accuracy improvement over state-of-the-arts on graph OOD generalization benchmarks.
  arXiv  Detail & Related papers  (2024-02-18T07:49:22Z)
• Elastic Information Bottleneck [34.90040361806197]
  Information bottleneck is an information-theoretic principle of representation learning. We propose an elastic information bottleneck (EIB) to interpolate between the IB and DIB regularizers. simulations and real data experiments show that EIB has the ability to achieve better domain adaptation results than IB and DIB.
  arXiv  Detail & Related papers  (2023-11-07T12:53:55Z)
• Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges [132.62934175555145]
  Neural Collapse (NC) is a well-known phenomenon of deep neural networks in the terminal phase of training (TPT) We propose a theoretical explanation for why continuing training can still lead to accuracy improvement on test set, even after the train accuracy has reached 100%. We refer to this newly discovered property as "non-conservative generalization"
  arXiv  Detail & Related papers  (2023-10-12T14:29:02Z)
• Recognizable Information Bottleneck [31.993478081354958]
  Information Bottlenecks (IBs) learn representations that generalize to unseen data by information compression. IBs are practically unable to guarantee generalization in real-world scenarios due to the vacuous generalization bound. We propose a Recognizable Information Bottleneck (RIB) which regularizes the recognizability of representations through a recognizability critic.
  arXiv  Detail & Related papers  (2023-04-28T03:55:33Z)
• Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspective [25.157282476221482]
  We show that the PAC-Bayesian theory can provide an explicit understanding of the relationship between LTH and generalization behavior. We offer the PAC-Bayes bound using a spike-and-slab distribution to analyze winning tickets.
  arXiv  Detail & Related papers  (2022-05-15T15:58:27Z)
• Gated Information Bottleneck for Generalization in Sequential Environments [13.795129636387623]
  Deep neural networks suffer from poor generalization to unseen environments when the underlying data distribution is different from that in the training set. We propose a new neural network-based IB approach, termed gated information bottleneck (GIB) We empirically demonstrate the superiority of GIB over other popular neural network-based IB approaches in adversarial robustness and out-of-distribution detection.
  arXiv  Detail & Related papers  (2021-10-12T14:58:38Z)
• A PAC-Bayesian Approach to Generalization Bounds for Graph Neural Networks [99.46182575751271]
  We derive generalization bounds for the two primary classes of graph neural networks (GNNs) Our result reveals that the maximum node degree and spectral norm of the weights govern the generalization bounds of both models.
  arXiv  Detail & Related papers  (2020-12-14T16:41:23Z)
• An Investigation into the Stochasticity of Batch Whitening [95.54842420166862]
  This paper investigates the more general Batch Whitening (BW) operation. We show that while various whitening transformations equivalently improve the conditioning, they show significantly different behaviors in discriminative scenarios and training Generative Adrial Networks (GAN) Our proposed BW algorithm improves the residual networks by a significant margin on ImageNetversaity.
  arXiv  Detail & Related papers  (2020-03-27T11:06:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.