Separable Computation of Information Measures

• URL: http://arxiv.org/abs/2501.15301v1
• Date: Sat, 25 Jan 2025 18:53:55 GMT
• Title: Separable Computation of Information Measures
• Authors: Xiangxiang Xu, Lizhong Zheng,
• Abstract summary: We study a separable design for computing information measures, where the information measure is computed from learned feature representations instead of raw data. We show that a class of information measures admit such separable computation, including mutual information, $f$-information, Wyner's common information, G'acs--K"orner common information, and Tishby's information bottleneck.
• Score: 5.807950618412389
• License:
• Abstract: We study a separable design for computing information measures, where the information measure is computed from learned feature representations instead of raw data. Under mild assumptions on the feature representations, we demonstrate that a class of information measures admit such separable computation, including mutual information, $f$-information, Wyner's common information, G{\'a}cs--K{\"o}rner common information, and Tishby's information bottleneck. Our development establishes several new connections between information measures and the statistical dependence structure. The characterizations also provide theoretical guarantees of practical designs for estimating information measures through representation learning.

Related papers

• Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training [14.9343236333741]
  We utilize information-theoretic metrics like matrix entropy and mutual information to analyze supervised learning. We show that matrix entropy cannot solely describe the interaction of the information content of data representation and classification head weights but it can effectively reflect the similarity and clustering behavior of the data.
  arXiv  Detail & Related papers  (2024-09-25T09:26:06Z)
• Comparing the information content of probabilistic representation spaces [3.7277730514654555]
  Probabilistic representation spaces convey information about a dataset and are shaped by factors such as the training data, network architecture, and loss function. We propose two information-theoretic measures to compare general probabilistic representation spaces. We demonstrate the utility of these measures in three case studies.
  arXiv  Detail & Related papers  (2024-05-31T17:33:07Z)
• Information Plane Analysis Visualization in Deep Learning via Transfer Entropy [0.0]
  In a feedforward network, Transfer Entropy can be used to measure the influence that one layer has on another. In contrast to mutual information, TE can capture temporal relationships between variables.
  arXiv  Detail & Related papers  (2024-04-01T17:34:18Z)
• Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective [68.20531518525273]
  We take a closer look into existing self-supervised methods of speech from an information-theoretic perspective. We use linear probes to estimate the mutual information between the target information and learned representations. We explore the potential of evaluating representations in a self-supervised fashion, where we estimate the mutual information between different parts of the data without using any labels.
  arXiv  Detail & Related papers  (2024-01-16T21:13:22Z)
• Information-Theoretic Odometry Learning [83.36195426897768]
  We propose a unified information theoretic framework for learning-motivated methods aimed at odometry estimation. The proposed framework provides an elegant tool for performance evaluation and understanding in information-theoretic language.
  arXiv  Detail & Related papers  (2022-03-11T02:37:35Z)
• Information-Bottleneck-Based Behavior Representation Learning for Multi-agent Reinforcement learning [16.024781473545055]
  In deep reinforcement learning, extracting sufficient and compact information of other agents is critical to attain efficient convergence and scalability of an algorithm. We present Information-Bottleneck-based Other agents' behavior Representation learning for Multi-agent reinforcement learning (IBORM) to explicitly seek low-dimensional mapping encoder.
  arXiv  Detail & Related papers  (2021-09-29T04:22:49Z)
• Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization [77.8735802150511]
  We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator. We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-08-11T21:17:02Z)
• Integrating Auxiliary Information in Self-supervised Learning [94.11964997622435]
  We first observe that the auxiliary information may bring us useful information about data structures. We present to construct data clusters according to the auxiliary information. We show that Cl-InfoNCE may be a better approach to leverage the data clustering information.
  arXiv  Detail & Related papers  (2021-06-05T11:01:15Z)
• Estimating informativeness of samples with Smooth Unique Information [108.25192785062367]
  We measure how much a sample informs the final weights and how much it informs the function computed by the weights. We give efficient approximations of these quantities using a linearized network. We apply these measures to several problems, such as dataset summarization.
  arXiv  Detail & Related papers  (2021-01-17T10:29:29Z)
• A Theory of Usable Information Under Computational Constraints [103.5901638681034]
  We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory. We show that by incorporating computational constraints, $mathcalV$-information can be reliably estimated from data.
  arXiv  Detail & Related papers  (2020-02-25T06:09:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.