MMG: Mutual Information Estimation via the MMSE Gap in Diffusion

• URL: http://arxiv.org/abs/2509.20609v1
• Date: Wed, 24 Sep 2025 23:04:48 GMT
• Title: MMG: Mutual Information Estimation via the MMSE Gap in Diffusion
• Authors: Longxuan Yu, Xing Shi, Xianghao Kong, Tong Jia, Greg Ver Steeg,
• Abstract summary: Mutual information (MI) is one of the most general ways to measure relationships between random variables.<n>Denoising diffusion models have recently set a new bar for density estimation.<n>We show the diffusion models can be used in a straightforward way to estimate MI.
• Score: 25.691925207007795
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mutual information (MI) is one of the most general ways to measure relationships between random variables, but estimating this quantity for complex systems is challenging. Denoising diffusion models have recently set a new bar for density estimation, so it is natural to consider whether these methods could also be used to improve MI estimation. Using the recently introduced information-theoretic formulation of denoising diffusion models, we show the diffusion models can be used in a straightforward way to estimate MI. In particular, the MI corresponds to half the gap in the Minimum Mean Square Error (MMSE) between conditional and unconditional diffusion, integrated over all Signal-to-Noise-Ratios (SNRs) in the noising process. Our approach not only passes self-consistency tests but also outperforms traditional and score-based diffusion MI estimators. Furthermore, our method leverages adaptive importance sampling to achieve scalable MI estimation, while maintaining strong performance even when the MI is high.

Related papers

• Score-based Membership Inference on Diffusion Models [3.742113529511043]
  Membership inference attacks (MIAs) against diffusion models have emerged as a pressing privacy concern.<n>We present a theoretical and empirical study of score-based MIAs, focusing on the predicted noise vectors that diffusion models learn to approximate.<n>We show that the expected denoiser output points toward a kernel-weighted local mean of nearby training samples, such that its norm encodes proximity to the training set and thereby reveals membership.
  arXiv  Detail & Related papers  (2025-09-29T16:28:55Z)
• A Deep Bayesian Nonparametric Framework for Robust Mutual Information Estimation [9.68824512279232]
  Mutual Information (MI) is a crucial measure for capturing dependencies between variables.<n>We present a solution for training an MI estimator by constructing the MI loss with a finite representation of the Dirichlet process posterior to incorporate regularization.<n>We explore the application of our estimator in maximizing MI between the data space and the latent space of a variational autoencoder.
  arXiv  Detail & Related papers  (2025-03-11T21:27:48Z)
• One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls [77.42510898755037]
  One More Step (OMS) is a compact network that incorporates an additional simple yet effective step during inference. OMS elevates image fidelity and harmonizes the dichotomy between training and inference, while preserving original model parameters. Once trained, various pre-trained diffusion models with the same latent domain can share the same OMS module.
  arXiv  Detail & Related papers  (2023-11-27T12:02:42Z)
• MINDE: Mutual Information Neural Diffusion Estimation [7.399561232927219]
  We present a new method for the estimation of Mutual Information (MI) between random variables. We use score-based diffusion models to estimate the Kullback Leibler divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables.
  arXiv  Detail & Related papers  (2023-10-13T11:47:41Z)
• Reconstructing Graph Diffusion History from a Single Snapshot [87.20550495678907]
  We propose a novel barycenter formulation for reconstructing Diffusion history from A single SnapsHot (DASH) We prove that estimation error of diffusion parameters is unavoidable due to NP-hardness of diffusion parameter estimation. We also develop an effective solver named DIffusion hiTting Times with Optimal proposal (DITTO)
  arXiv  Detail & Related papers  (2023-06-01T09:39:32Z)
• Improving Mutual Information Estimation with Annealed and Energy-Based Bounds [20.940022170594816]
  Mutual information (MI) is a fundamental quantity in information theory and machine learning. We present a unifying view of existing MI bounds from the perspective of importance sampling. We propose three novel bounds based on this approach.
  arXiv  Detail & Related papers  (2023-03-13T10:47:24Z)
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models [76.76860707897413]
  Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution. We show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process.
  arXiv  Detail & Related papers  (2022-06-10T15:09:46Z)
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization [69.07420650261649]
  We introduce a novel, simple, and powerful contrastive MI estimator named as FLO. Empirically, our FLO estimator overcomes the limitations of its predecessors and learns more efficiently. The utility of FLO is verified using an extensive set of benchmarks, which also reveals the trade-offs in practical MI estimation.
  arXiv  Detail & Related papers  (2021-07-02T15:20:41Z)
• Reducing the Variance of Variational Estimates of Mutual Information by Limiting the Critic's Hypothesis Space to RKHS [0.0]
  Mutual information (MI) is an information-theoretic measure of dependency between two random variables. Recent methods realize parametric probability distributions or critic as a neural network to approximate unknown density ratios. We argue that the high variance characteristic is due to the uncontrolled complexity of the critic's hypothesis space.
  arXiv  Detail & Related papers  (2020-11-17T14:32:48Z)
• A Rigorous Link Between Self-Organizing Maps and Gaussian Mixture Models [78.6363825307044]
  This work presents a mathematical treatment of the relation between Self-Organizing Maps (SOMs) and Gaussian Mixture Models (GMMs) We show that energy-based SOM models can be interpreted as performing gradient descent. This link allows to treat SOMs as generative probabilistic models, giving a formal justification for using SOMs to detect outliers, or for sampling.
  arXiv  Detail & Related papers  (2020-09-24T14:09:04Z)
• Neural Methods for Point-wise Dependency Estimation [129.93860669802046]
  We focus on estimating point-wise dependency (PD), which quantitatively measures how likely two outcomes co-occur. We demonstrate the effectiveness of our approaches in 1) MI estimation, 2) self-supervised representation learning, and 3) cross-modal retrieval task.
  arXiv  Detail & Related papers  (2020-06-09T23:26:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.