Elastic Information Bottleneck

• URL: http://arxiv.org/abs/2311.03955v1
• Date: Tue, 7 Nov 2023 12:53:55 GMT
• Title: Elastic Information Bottleneck
• Authors: Yuyan Ni, Yanyan Lan, Ao Liu, Zhiming Ma
• Abstract summary: Information bottleneck is an information-theoretic principle of representation learning. We propose an elastic information bottleneck (EIB) to interpolate between the IB and DIB regularizers. simulations and real data experiments show that EIB has the ability to achieve better domain adaptation results than IB and DIB.
• Score: 34.90040361806197
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Information bottleneck is an information-theoretic principle of representation learning that aims to learn a maximally compressed representation that preserves as much information about labels as possible. Under this principle, two different methods have been proposed, i.e., information bottleneck (IB) and deterministic information bottleneck (DIB), and have gained significant progress in explaining the representation mechanisms of deep learning algorithms. However, these theoretical and empirical successes are only valid with the assumption that training and test data are drawn from the same distribution, which is clearly not satisfied in many real-world applications. In this paper, we study their generalization abilities within a transfer learning scenario, where the target error could be decomposed into three components, i.e., source empirical error, source generalization gap (SG), and representation discrepancy (RD). Comparing IB and DIB on these terms, we prove that DIB's SG bound is tighter than IB's while DIB's RD is larger than IB's. Therefore, it is difficult to tell which one is better. To balance the trade-off between SG and the RD, we propose an elastic information bottleneck (EIB) to interpolate between the IB and DIB regularizers, which guarantees a Pareto frontier within the IB framework. Additionally, simulations and real data experiments show that EIB has the ability to achieve better domain adaptation results than IB and DIB, which validates the correctness of our theories.

Related papers

• Disentangled Representation Learning with Transmitted Information Bottleneck [57.22757813140418]
  We present textbfDisTIB (textbfTransmitted textbfInformation textbfBottleneck for textbfDisd representation learning), a novel objective that navigates the balance between information compression and preservation.
  arXiv  Detail & Related papers  (2023-11-03T03:18:40Z)
• Combating Bilateral Edge Noise for Robust Link Prediction [56.43882298843564]
  We propose an information-theory-guided principle, Robust Graph Information Bottleneck (RGIB), to extract reliable supervision signals and avoid representation collapse. Two instantiations, RGIB-SSL and RGIB-REP, are explored to leverage the merits of different methodologies. Experiments on six datasets and three GNNs with diverse noisy scenarios verify the effectiveness of our RGIB instantiations.
  arXiv  Detail & Related papers  (2023-11-02T12:47:49Z)
• Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalization [60.4169201192582]
  We propose to incorporate label and environment causal independence (LECI) to fully make use of label and environment information. LECI significantly outperforms prior methods on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-06-01T19:33:30Z)
• Recognizable Information Bottleneck [31.993478081354958]
  Information Bottlenecks (IBs) learn representations that generalize to unseen data by information compression. IBs are practically unable to guarantee generalization in real-world scenarios due to the vacuous generalization bound. We propose a Recognizable Information Bottleneck (RIB) which regularizes the recognizability of representations through a recognizability critic.
  arXiv  Detail & Related papers  (2023-04-28T03:55:33Z)
• Variational Distillation for Multi-View Learning [104.17551354374821]
  We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning. Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
  arXiv  Detail & Related papers  (2022-06-20T03:09:46Z)
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control? [80.2534918595143]
  Mutual information provides an appealing formalism for learning representations of data. This paper formalizes the sufficiency of a state representation for learning and representing the optimal policy. Surprisingly, we find that two of these objectives can yield insufficient representations given mild and common assumptions on the structure of the MDP.
  arXiv  Detail & Related papers  (2021-06-14T10:12:34Z)
• Perturbation Theory for the Information Bottleneck [6.117084972237769]
  Information bottleneck (IB) method formalizes extracting relevant information from data. nonlinearity of the IB problem makes it computationally expensive and analytically intractable in general. We derive a perturbation theory for the IB method and report the first complete characterization of the learning onset.
  arXiv  Detail & Related papers  (2021-05-28T16:59:01Z)
• Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification [41.02729491273057]
  The Information Bottleneck (IB) provides an information theoretic principle for representation learning. We present a new strategy, Variational Self-Distillation (VSD), which provides a scalable, flexible and analytic solution. We also introduce two other strategies, Variational Cross-Distillation (VCD) and Variational Mutual-Learning (VML)
  arXiv  Detail & Related papers  (2021-04-07T02:19:41Z)
• The Dual Information Bottleneck [1.6559345531428509]
  The Information Bottleneck (IB) framework is a general characterization of optimal representations obtained using a principled approach for balancing accuracy and complexity. We present a new framework, the Dual Information Bottleneck (dualIB) which resolves some of the known drawbacks of the IB.
  arXiv  Detail & Related papers  (2020-06-08T14:43:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.