Information Subtraction: Learning Representations for Conditional Entropy

• URL: http://arxiv.org/abs/2501.02012v1
• Date: Thu, 02 Jan 2025 13:10:31 GMT
• Title: Information Subtraction: Learning Representations for Conditional Entropy
• Authors: Keng Hou Leong, Yuxuan Xiu, Wai Kin, Chan,
• Abstract summary: This paper introduces Information Subtraction, a framework designed to generate representations that preserve desired information while eliminating the undesired. We implement a generative-based architecture that outputs these representations by simultaneously maximizing an information term and minimizing another. Our results highlight the representations' ability to provide semantic features of conditional entropy.
• Score: 1.4297089600426414
• License:
• Abstract: The representations of conditional entropy and conditional mutual information are significant in explaining the unique effects among variables. While previous studies based on conditional contrastive sampling have effectively removed information regarding discrete sensitive variables, they have not yet extended their scope to continuous cases. This paper introduces Information Subtraction, a framework designed to generate representations that preserve desired information while eliminating the undesired. We implement a generative-based architecture that outputs these representations by simultaneously maximizing an information term and minimizing another. With its flexibility in disentangling information, we can iteratively apply Information Subtraction to represent arbitrary information components between continuous variables, thereby explaining the various relationships that exist between them. Our results highlight the representations' ability to provide semantic features of conditional entropy. By subtracting sensitive and domain-specific information, our framework demonstrates effective performance in fair learning and domain generalization. The code for this paper is available at https://github.com/jh-liang/Information-Subtraction

Related papers

• CSTA: Spatial-Temporal Causal Adaptive Learning for Exemplar-Free Video Class-Incremental Learning [62.69917996026769]
  A class-incremental learning task requires learning and preserving both spatial appearance and temporal action involvement. We propose a framework that equips separate adapters to learn new class patterns, accommodating the incremental information requirements unique to each class. A causal compensation mechanism is proposed to reduce the conflicts during increment and memorization for between different types of information.
  arXiv  Detail & Related papers  (2025-01-13T11:34:55Z)
• Representation Learning with Conditional Information Flow Maximization [29.36409607847339]
  This paper proposes an information-theoretic representation learning framework, named conditional information flow. It promotes learned representations have good feature uniformity and sufficient predictive ability. Experiments show that the learned representations are more sufficient, robust and transferable.
  arXiv  Detail & Related papers  (2024-06-08T16:19:18Z)
• Efficient Information Extraction in Few-Shot Relation Classification through Contrastive Representation Learning [23.992247765851204]
  We introduce a novel approach to enhance information extraction combining multiple sentence representations and contrastive learning. Our method employs contrastive learning to extract complementary discriminative information from these individual representations.
  arXiv  Detail & Related papers  (2024-03-25T08:36:06Z)
• Ignorance is Bliss: Robust Control via Information Gating [60.17644038829572]
  Informational parsimony provides a useful inductive bias for learning representations that achieve better generalization by being robust to noise and spurious correlations. We propose textitinformation gating as a way to learn parsimonious representations that identify the minimal information required for a task.
  arXiv  Detail & Related papers  (2023-03-10T18:31:50Z)
• FUNCK: Information Funnels and Bottlenecks for Invariant Representation Learning [7.804994311050265]
  We investigate a set of related information funnels and bottleneck problems that claim to learn invariant representations from the data. We propose a new element to this family of information-theoretic objectives: The Conditional Privacy Funnel with Side Information. Given the generally intractable objectives, we derive tractable approximations using amortized variational inference parameterized by neural networks.
  arXiv  Detail & Related papers  (2022-11-02T19:37:55Z)
• Leveraging Relational Information for Learning Weakly Disentangled Representations [11.460692362624533]
  Disentanglement is a difficult property to enforce in neural representations. We present an alternative view over learning (weakly) disentangled representations.
  arXiv  Detail & Related papers  (2022-05-20T09:58:51Z)
• Learning Conditional Invariance through Cycle Consistency [60.85059977904014]
  We propose a novel approach to identify meaningful and independent factors of variation in a dataset. Our method involves two separate latent subspaces for the target property and the remaining input information. We demonstrate on synthetic and molecular data that our approach identifies more meaningful factors which lead to sparser and more interpretable models.
  arXiv  Detail & Related papers  (2021-11-25T17:33:12Z)
• Conditional Contrastive Learning: Removing Undesirable Information in Self-Supervised Representations [108.29288034509305]
  We develop conditional contrastive learning to remove undesirable information in self-supervised representations. We demonstrate empirically that our methods can successfully learn self-supervised representations for downstream tasks.
  arXiv  Detail & Related papers  (2021-06-05T10:51:26Z)
• Fairness by Learning Orthogonal Disentangled Representations [50.82638766862974]
  We propose a novel disentanglement approach to invariant representation problem. We enforce the meaningful representation to be agnostic to sensitive information by entropy. The proposed approach is evaluated on five publicly available datasets.
  arXiv  Detail & Related papers  (2020-03-12T11:09:15Z)
• A Theory of Usable Information Under Computational Constraints [103.5901638681034]
  We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory. We show that by incorporating computational constraints, $mathcalV$-information can be reliably estimated from data.
  arXiv  Detail & Related papers  (2020-02-25T06:09:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.