Beyond Normal: On the Evaluation of Mutual Information Estimators

• URL: http://arxiv.org/abs/2306.11078v2
• Date: Mon, 16 Oct 2023 13:17:17 GMT
• Title: Beyond Normal: On the Evaluation of Mutual Information Estimators
• Authors: Pawe{\l} Czy\.z, Frederic Grabowski, Julia E. Vogt, Niko Beerenwinkel, Alexander Marx
• Abstract summary: We show how to construct a diverse family of distributions with known ground-truth mutual information. We provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered.
• Score: 52.85079110699378
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Mutual information is a general statistical dependency measure which has found applications in representation learning, causality, domain generalization and computational biology. However, mutual information estimators are typically evaluated on simple families of probability distributions, namely multivariate normal distribution and selected distributions with one-dimensional random variables. In this paper, we show how to construct a diverse family of distributions with known ground-truth mutual information and propose a language-independent benchmarking platform for mutual information estimators. We discuss the general applicability and limitations of classical and neural estimators in settings involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. Finally, we provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered and issues one needs to consider when applying an estimator to a new data set.

Related papers

• Mutual Information Multinomial Estimation [53.58005108981247]
  Estimating mutual information (MI) is a fundamental yet challenging task in data science and machine learning. Our main discovery is that a preliminary estimate of the data distribution can dramatically help estimate. Experiments on diverse tasks including non-Gaussian synthetic problems with known ground-truth and real-world applications demonstrate the advantages of our method.
  arXiv  Detail & Related papers  (2024-08-18T06:27:30Z)
• On the Properties and Estimation of Pointwise Mutual Information Profiles [49.877314063833296]
  The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. We introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods.
  arXiv  Detail & Related papers  (2023-10-16T10:02:24Z)
• Variational $f$-Divergence and Derangements for Discriminative Mutual Information Estimation [4.114444605090134]
  We propose a novel class of discriminative mutual information estimators based on the variational representation of the $f$-divergence. Experiments on reference scenarios demonstrate that our approach outperforms state-of-the-art neural estimators both in terms of accuracy and complexity.
  arXiv  Detail & Related papers  (2023-05-31T16:54:25Z)
• Gacs-Korner Common Information Variational Autoencoder [102.89011295243334]
  We propose a notion of common information that allows one to quantify and separate the information that is shared between two random variables. We demonstrate that our formulation allows us to learn semantically meaningful common and unique factors of variation even on high-dimensional data such as images and videos.
  arXiv  Detail & Related papers  (2022-05-24T17:47:26Z)
• Interaction Models and Generalized Score Matching for Compositional Data [9.797319790710713]
  We propose a class of exponential family models that accommodate general patterns of pairwise interaction while being supported on the probability simplex. Special cases include the family of Dirichlet distributions as well as Aitchison's additive logistic normal distributions. A high-dimensional analysis of our estimation methods shows that the simplex domain is handled as efficiently as previously studied full-dimensional domains.
  arXiv  Detail & Related papers  (2021-09-10T05:29:41Z)
• Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization [77.8735802150511]
  We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator. We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-08-11T21:17:02Z)
• An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization [53.592597682854944]
  We recast generalization over sub-groups as an online game between a player minimizing risk and an adversary presenting new test. We show that ERM is provably minimax-optimal for both tasks.
  arXiv  Detail & Related papers  (2021-02-25T19:06:48Z)
• DEMI: Discriminative Estimator of Mutual Information [5.248805627195347]
  Estimating mutual information between continuous random variables is often intractable and challenging for high-dimensional data. Recent progress has leveraged neural networks to optimize variational lower bounds on mutual information. Our approach is based on training a classifier that provides the probability that a data sample pair is drawn from the joint distribution.
  arXiv  Detail & Related papers  (2020-10-05T04:19:27Z)
• Learning Unbiased Representations via Mutual Information Backpropagation [36.383338079229695]
  In particular, we face the case where some attributes (bias) of the data, if learned by the model, can severely compromise its generalization properties. We propose a novel end-to-end optimization strategy, which simultaneously estimates and minimizes the mutual information between the learned representation and the data attributes.
  arXiv  Detail & Related papers  (2020-03-13T18:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.