InfoBridge: Mutual Information estimation via Bridge Matching

• URL: http://arxiv.org/abs/2502.01383v1
• Date: Mon, 03 Feb 2025 14:18:37 GMT
• Title: InfoBridge: Mutual Information estimation via Bridge Matching
• Authors: Sergei Kholkin, Ivan Butakov, Evgeny Burnaev, Nikita Gushchin, Alexander Korotin,
• Abstract summary: We show that by using the theory of diffusion bridges, one can construct an unbiased estimator for data posing difficulties for conventional MI estimators. We showcase the performance of our estimator on a series of standard MI estimation benchmarks.
• Score: 64.11574776911542
• License:
• Abstract: Diffusion bridge models have recently become a powerful tool in the field of generative modeling. In this work, we leverage their power to address another important problem in machine learning and information theory - the estimation of the mutual information (MI) between two random variables. We show that by using the theory of diffusion bridges, one can construct an unbiased estimator for data posing difficulties for conventional MI estimators. We showcase the performance of our estimator on a series of standard MI estimation benchmarks.

Related papers

• A Neural Difference-of-Entropies Estimator for Mutual Information [2.3020018305241337]
  We propose a novel mutual information estimator based on parametrizing conditional densities using normalizing flows. This estimator leverages a block autoregressive structure to achieve improved bias-variance trade-offs on standard benchmark tasks.
  arXiv  Detail & Related papers  (2025-02-18T17:48:25Z)
• Mutual Information Multinomial Estimation [53.58005108981247]
  Estimating mutual information (MI) is a fundamental yet challenging task in data science and machine learning. Our main discovery is that a preliminary estimate of the data distribution can dramatically help estimate. Experiments on diverse tasks including non-Gaussian synthetic problems with known ground-truth and real-world applications demonstrate the advantages of our method.
  arXiv  Detail & Related papers  (2024-08-18T06:27:30Z)
• Diffusion posterior sampling for simulation-based inference in tall data settings [53.17563688225137]
  Simulation-based inference ( SBI) is capable of approximating the posterior distribution that relates input parameters to a given observation. In this work, we consider a tall data extension in which multiple observations are available to better infer the parameters of the model. We compare our method to recently proposed competing approaches on various numerical experiments and demonstrate its superiority in terms of numerical stability and computational cost.
  arXiv  Detail & Related papers  (2024-04-11T09:23:36Z)
• Mutual Information Estimation via Normalizing Flows [39.58317527488534]
  We propose a novel approach to the problem of mutual information estimation. The estimator maps original data to the target distribution, for which MI is easier to estimate. We additionally explore the target distributions with known closed-form expressions for MI.
  arXiv  Detail & Related papers  (2024-03-04T16:28:04Z)
• Debiasing Multimodal Models via Causal Information Minimization [65.23982806840182]
  We study bias arising from confounders in a causal graph for multimodal data. Robust predictive features contain diverse information that helps a model generalize to out-of-distribution data. We use these features as confounder representations and use them via methods motivated by causal theory to remove bias from models.
  arXiv  Detail & Related papers  (2023-11-28T16:46:14Z)
• Time Series Anomaly Detection using Diffusion-based Models [5.896413260185387]
  Diffusion models have been recently used for anomaly detection in images. We investigate whether they can also be leveraged for anomaly detection on multivariate time series. Our models outperform the baselines on synthetic datasets and are competitive on real-world datasets.
  arXiv  Detail & Related papers  (2023-11-02T17:58:09Z)
• Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models [77.83923746319498]
  We propose a framework called Diff-Instruct to instruct the training of arbitrary generative models. We show that Diff-Instruct results in state-of-the-art single-step diffusion-based models. Experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models.
  arXiv  Detail & Related papers  (2023-05-29T04:22:57Z)
• ER: Equivariance Regularizer for Knowledge Graph Completion [107.51609402963072]
  We propose a new regularizer, namely, Equivariance Regularizer (ER) ER can enhance the generalization ability of the model by employing the semantic equivariance between the head and tail entities. The experimental results indicate a clear and substantial improvement over the state-of-the-art relation prediction methods.
  arXiv  Detail & Related papers  (2022-06-24T08:18:05Z)
• The Conditional Entropy Bottleneck [8.797368310561058]
  We characterize failures of robust generalization as failures of accuracy or related metrics on a held-out set. We propose the Minimum Necessary Information (MNI) criterion for evaluating the quality of a model. In order to train models that perform well with respect to the MNI criterion, we present a new objective function, the Conditional Entropy Bottleneck (CEB) We experimentally test our hypothesis by comparing the performance of CEB models with deterministic models and Variational Information Bottleneck (VIB) models on a variety of different datasets.
  arXiv  Detail & Related papers  (2020-02-13T07:46:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.