NMINE: Normalized Mutual Information Neural Estimation

• URL: http://arxiv.org/abs/2607.27710v1
• Date: Thu, 30 Jul 2026 05:44:34 GMT
• Title: NMINE: Normalized Mutual Information Neural Estimation
• Abstract summary: We propose a fully neural normalized mutual information estimator for continuous variables.<n> Mutual information is estimated using the Donsker--Varadhan representation, while marginal entropies are estimated by learning the divergence between each marginal distribution and a uniform reference distribution.<n> Experiments on Gaussian data from one to eight dimensions show that the proposed estimator improves accuracy over a KSG-based normalized mutual information baseline.
• Score: 7.18961038438762
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mutual information is a general measure of statistical dependence that captures both linear and nonlinear relationships between random variables. For continuous and multidimensional variables For continuous multidimensional variables, mutual information must be estimated from samples. Because mutual information is unbounded, its values are not directly comparable across datasets, dimensions, or applications. Normalized mutual information addresses this limitation by converting mutual information into a normalized dependency score. Recent work has demonstrated the practical value of normalized mutual information in applications such as molecular dynamics {arXiv:2405.04980} and interpretable machine learning {arXiv:2409.16768}, but existing estimators remain sensitive to dimensionality and numerical stability {arXiv:2410.07642}. In this paper, we propose a fully neural normalized mutual information estimator for continuous variables. The proposed approach combines a MINE-based neural mutual information estimator {arXiv:1801.04062} with MI-NEE-inspired neural marginal entropy estimators {arXiv:1905.12957}. Mutual information is estimated using the Donsker--Varadhan representation, while marginal entropies are estimated by learning the divergence between each marginal distribution and a uniform reference distribution, from which entropy is recovered. The resulting estimator provides a neural alternative to k-nearest-neighbor-based normalized mutual information estimation {arXiv:2405.04980}. Experiments on Gaussian data from one to eight dimensions show that the proposed estimator improves accuracy over a KSG-based normalized mutual information baseline. These results indicate that neural estimation is a promising direction for normalized dependency measurement in continuous multidimensional settings.

Related papers

• InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate [68.14572758582081]
  InfoAtlas is a foundation model-like architecture that eliminates the need for iterative optimization.<n>It learns to identify diverse dependence structures and predict MI directly from the dataset.<n>Experiments demonstrate that InfoAtlas matches state-of-the-art neural estimators in accuracy while achieving $100times$ speedup.
  arXiv  Detail & Related papers  (2026-05-29T18:16:51Z)
• Efficient Covariance Estimation for Sparsified Functional Data [51.69796254617083]
  proposed Random-knots (Random-knots-Spatial) and B-spline (Bspline-Spatial) estimators of the covariance function are computationally efficient.<n>Asymptotic pointwise of the covariance are obtained for sparsified individual trajectories under some regularity conditions.
  arXiv  Detail & Related papers  (2025-11-23T00:50:33Z)
• A Neural Difference-of-Entropies Estimator for Mutual Information [1.3714815350300447]
  We propose a novel mutual information estimator based on parametrizing conditional densities using normalizing flows.<n>This estimator leverages a block autoregressive structure to achieve improved bias-variance trade-offs on standard benchmark tasks.
  arXiv  Detail & Related papers  (2025-02-18T17:48:25Z)
• Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
  We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data. Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
  arXiv  Detail & Related papers  (2024-04-24T09:04:36Z)
• On the Properties and Estimation of Pointwise Mutual Information Profiles [49.877314063833296]
  The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. We introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods.
  arXiv  Detail & Related papers  (2023-10-16T10:02:24Z)
• Max-Sliced Mutual Information [17.667315953598788]
  Quantifying the dependence between high-dimensional random variables is central to statistical learning and inference. Two classical methods are canonical correlation analysis (CCA), which identifies maximally correlated projected versions of the original variables, and Shannon's mutual information, which is a universal dependence measure. This work proposes a middle ground in the form of a scalable information-theoretic generalization of CCA, termed max-sliced mutual information (mSMI)
  arXiv  Detail & Related papers  (2023-09-28T06:49:25Z)
• Beyond Normal: On the Evaluation of Mutual Information Estimators [52.85079110699378]
  We show how to construct a diverse family of distributions with known ground-truth mutual information. We provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered.
  arXiv  Detail & Related papers  (2023-06-19T17:26:34Z)
• Information Theory Measures via Multidimensional Gaussianization [7.788961560607993]
  Information theory is an outstanding framework to measure uncertainty, dependence and relevance in data and systems. It has several desirable properties for real world applications. However, obtaining information from multidimensional data is a challenging problem due to the curse of dimensionality.
  arXiv  Detail & Related papers  (2020-10-08T07:22:16Z)
• Parametric Copula-GP model for analyzing multidimensional neuronal and behavioral relationships [2.624902795082451]
  We propose a parametric copula model which separates the statistics of the individual variables from their dependence structure. We use a Bayesian framework with Gaussian Process (GP) priors over copula parameters, conditioned on a continuous task-related variable. Our framework is particularly useful for the analysis of complex multidimensional relationships between neuronal, sensory and behavioral data.
  arXiv  Detail & Related papers  (2020-08-03T16:44:29Z)
• Neural Methods for Point-wise Dependency Estimation [129.93860669802046]
  We focus on estimating point-wise dependency (PD), which quantitatively measures how likely two outcomes co-occur. We demonstrate the effectiveness of our approaches in 1) MI estimation, 2) self-supervised representation learning, and 3) cross-modal retrieval task.
  arXiv  Detail & Related papers  (2020-06-09T23:26:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.