MINDE: Mutual Information Neural Diffusion Estimation

• URL: http://arxiv.org/abs/2310.09031v2
• Date: Wed, 15 May 2024 09:21:41 GMT
• Title: MINDE: Mutual Information Neural Diffusion Estimation
• Authors: Giulio Franzese, Mustapha Bounoua, Pietro Michiardi,
• Abstract summary: We present a new method for the estimation of Mutual Information (MI) between random variables. We use score-based diffusion models to estimate the Kullback Leibler divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables.
• Score: 7.399561232927219
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to estimate the Kullback Leibler divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables. Armed with such building blocks, we present a general recipe to measure MI, which unfolds in two directions: one uses conditional diffusion process, whereas the other uses joint diffusion processes that allow simultaneous modelling of two random variables. Our results, which derive from a thorough experimental protocol over all the variants of our approach, indicate that our method is more accurate than the main alternatives from the literature, especially for challenging distributions. Furthermore, our methods pass MI self-consistency tests, including data processing and additivity under independence, which instead are a pain-point of existing methods.

Related papers

• Total Uncertainty Quantification in Inverse PDE Solutions Obtained with Reduced-Order Deep Learning Surrogate Models [50.90868087591973]
  We propose an approximate Bayesian method for quantifying the total uncertainty in inverse PDE solutions obtained with machine learning surrogate models. We test the proposed framework by comparing it with the iterative ensemble smoother and deep ensembling methods for a non-linear diffusion equation.
  arXiv  Detail & Related papers  (2024-08-20T19:06:02Z)
• Improving Probabilistic Diffusion Models With Optimal Covariance Matching [27.2761325416843]
  We introduce a novel method for learning the diagonal covariances. We show how our method can substantially enhance the sampling efficiency, recall rate and likelihood of both diffusion models and latent diffusion models.
  arXiv  Detail & Related papers  (2024-06-16T05:47:12Z)
• Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
  We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data. Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
  arXiv  Detail & Related papers  (2024-04-24T09:04:36Z)
• Improved off-policy training of diffusion samplers [93.66433483772055]
  We study the problem of training diffusion models to sample from a distribution with an unnormalized density or energy function. We benchmark several diffusion-structured inference methods, including simulation-based variational approaches and off-policy methods. Our results shed light on the relative advantages of existing algorithms while bringing into question some claims from past work.
  arXiv  Detail & Related papers  (2024-02-07T18:51:49Z)
• Score-based Source Separation with Applications to Digital Communication Signals [72.6570125649502]
  We propose a new method for separating superimposed sources using diffusion-based generative models. Motivated by applications in radio-frequency (RF) systems, we are interested in sources with underlying discrete nature. Our method can be viewed as a multi-source extension to the recently proposed score distillation sampling scheme.
  arXiv  Detail & Related papers  (2023-06-26T04:12:40Z)
• Positive definite nonparametric regression using an evolutionary algorithm with application to covariance function estimation [0.0]
  We propose a novel nonparametric regression framework for estimating covariance functions of stationary processes. Our method can impose positive definiteness, as well as isotropy and monotonicity, on the estimators. Our method provides more reliable estimates for long-range dependence.
  arXiv  Detail & Related papers  (2023-04-25T22:01:14Z)
• Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
  We study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
  arXiv  Detail & Related papers  (2022-04-07T17:59:19Z)
• Comparing two samples through stochastic dominance: a graphical approach [2.867517731896504]
  Non-deterministic measurements are common in real-world scenarios. We propose an alternative framework to visually compare two samples according to their estimated cumulative distribution functions.
  arXiv  Detail & Related papers  (2022-03-15T13:37:03Z)
• DEMI: Discriminative Estimator of Mutual Information [5.248805627195347]
  Estimating mutual information between continuous random variables is often intractable and challenging for high-dimensional data. Recent progress has leveraged neural networks to optimize variational lower bounds on mutual information. Our approach is based on training a classifier that provides the probability that a data sample pair is drawn from the joint distribution.
  arXiv  Detail & Related papers  (2020-10-05T04:19:27Z)
• Neural Methods for Point-wise Dependency Estimation [129.93860669802046]
  We focus on estimating point-wise dependency (PD), which quantitatively measures how likely two outcomes co-occur. We demonstrate the effectiveness of our approaches in 1) MI estimation, 2) self-supervised representation learning, and 3) cross-modal retrieval task.
  arXiv  Detail & Related papers  (2020-06-09T23:26:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.