Mutual information and the encoding of contingency tables

• URL: http://arxiv.org/abs/2405.05393v1
• Date: Wed, 8 May 2024 19:49:39 GMT
• Title: Mutual information and the encoding of contingency tables
• Authors: Maximilian Jerdee, Alec Kirkley, M. E. J. Newman,
• Abstract summary: Mutual information is commonly used as a measure of similarity between labelings of a given set of objects. As argued recently, the mutual information as conventionally defined can return biased results because it neglects the information cost of the so-called contingency table. In this paper we describe an improved method for encoding contingency tables that gives a substantially better bound in typical use cases.
• Score: 0.4779196219827508
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mutual information is commonly used as a measure of similarity between competing labelings of a given set of objects, for example to quantify performance in classification and community detection tasks. As argued recently, however, the mutual information as conventionally defined can return biased results because it neglects the information cost of the so-called contingency table, a crucial component of the similarity calculation. In principle the bias can be rectified by subtracting the appropriate information cost, leading to the modified measure known as the reduced mutual information, but in practice one can only ever compute an upper bound on this information cost, and the value of the reduced mutual information depends crucially on how good a bound is established. In this paper we describe an improved method for encoding contingency tables that gives a substantially better bound in typical use cases, and approaches the ideal value in the common case where the labelings are closely similar, as we demonstrate with extensive numerical results.

Related papers

• Optimised Storage for Datalog Reasoning [8.305527776204178]
  Materialisation facilitates Datalog reasoning by precomputing all consequences of the facts and the rules so that queries can be directly answered over the materialised facts. storing all materialised facts may be infeasible in practice, especially when the rules are complex and the given set of facts is large. We present a general framework that allows for the integration of such optimised storage schemes with standard materialisation algorithms.
  arXiv  Detail & Related papers  (2023-12-18T15:46:10Z)
• Obtaining Explainable Classification Models using Distributionally Robust Optimization [12.511155426574563]
  We study generalized linear models constructed using sets of feature value rules. An inherent trade-off exists between rule set sparsity and its prediction accuracy. We propose a new formulation to learn an ensemble of rule sets that simultaneously addresses these competing factors.
  arXiv  Detail & Related papers  (2023-11-03T15:45:34Z)
• Normalized mutual information is a biased measure for classification and community detection [0.4779196219827508]
  We argue that results returned by the normalized mutual information are biased for two reasons. We show that one's conclusions about which algorithm is best are significantly affected by the biases in the traditional mutual information.
  arXiv  Detail & Related papers  (2023-07-03T18:12:32Z)
• Beyond Normal: On the Evaluation of Mutual Information Estimators [52.85079110699378]
  We show how to construct a diverse family of distributions with known ground-truth mutual information. We provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered.
  arXiv  Detail & Related papers  (2023-06-19T17:26:34Z)
• Gacs-Korner Common Information Variational Autoencoder [102.89011295243334]
  We propose a notion of common information that allows one to quantify and separate the information that is shared between two random variables. We demonstrate that our formulation allows us to learn semantically meaningful common and unique factors of variation even on high-dimensional data such as images and videos.
  arXiv  Detail & Related papers  (2022-05-24T17:47:26Z)
• Information-Theoretic Bias Reduction via Causal View of Spurious Correlation [71.9123886505321]
  We propose an information-theoretic bias measurement technique through a causal interpretation of spurious correlation. We present a novel debiasing framework against the algorithmic bias, which incorporates a bias regularization loss. The proposed bias measurement and debiasing approaches are validated in diverse realistic scenarios.
  arXiv  Detail & Related papers  (2022-01-10T01:19:31Z)
• Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization [77.8735802150511]
  We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator. We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-08-11T21:17:02Z)
• Cost-Based Budget Active Learning for Deep Learning [0.9732863739456035]
  We propose a Cost-Based Bugdet Active Learning (CBAL) which considers the classification uncertainty as well as instance diversity in a population constrained by a budget. A principled approach based on the min-max is considered to minimize both the labeling and decision cost of the selected instances.
  arXiv  Detail & Related papers  (2020-12-09T17:42:44Z)
• Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries [74.28810048824519]
  We analyze the token alignments used by ROUGE and BERTScore to compare summaries. We argue that their scores largely cannot be interpreted as measuring information overlap.
  arXiv  Detail & Related papers  (2020-10-23T15:55:15Z)
• Type-augmented Relation Prediction in Knowledge Graphs [65.88395564516115]
  We propose a type-augmented relation prediction (TaRP) method, where we apply both the type information and instance-level information for relation prediction. Our proposed TaRP method achieves significantly better performance than state-of-the-art methods on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-16T21:14:18Z)
• Learning Unbiased Representations via Mutual Information Backpropagation [36.383338079229695]
  In particular, we face the case where some attributes (bias) of the data, if learned by the model, can severely compromise its generalization properties. We propose a novel end-to-end optimization strategy, which simultaneously estimates and minimizes the mutual information between the learned representation and the data attributes.
  arXiv  Detail & Related papers  (2020-03-13T18:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.