Convergence of Statistical Estimators via Mutual Information Bounds

• URL: http://arxiv.org/abs/2412.18539v1
• Date: Tue, 24 Dec 2024 16:42:45 GMT
• Title: Convergence of Statistical Estimators via Mutual Information Bounds
• Authors: El Mahdi Khribch, Pierre Alquier,
• Abstract summary: We introduce a novel mutual information bound for statistical models. The derived bound has wide-ranging applications in statistical inference. It can also be used to study a wide range of estimation methods.
• Score: 3.9041951045689305
• License:
• Abstract: Recent advances in statistical learning theory have revealed profound connections between mutual information (MI) bounds, PAC-Bayesian theory, and Bayesian nonparametrics. This work introduces a novel mutual information bound for statistical models. The derived bound has wide-ranging applications in statistical inference. It yields improved contraction rates for fractional posteriors in Bayesian nonparametrics. It can also be used to study a wide range of estimation methods, such as variational inference or Maximum Likelihood Estimation (MLE). By bridging these diverse areas, this work advances our understanding of the fundamental limits of statistical inference and the role of information in learning from data. We hope that these results will not only clarify connections between statistical inference and information theory but also help to develop a new toolbox to study a wide range of estimators.

Related papers

• Estimation of the Learning Coefficient Using Empirical Loss [0.9208007322096532]
  The learning coefficient plays a crucial role in analyzing the performance of information criteria. We propose a novel numerical estimation method that fundamentally differs from previous approaches.
  arXiv  Detail & Related papers  (2025-02-14T08:30:04Z)
• Mutual Information Multinomial Estimation [53.58005108981247]
  Estimating mutual information (MI) is a fundamental yet challenging task in data science and machine learning. Our main discovery is that a preliminary estimate of the data distribution can dramatically help estimate. Experiments on diverse tasks including non-Gaussian synthetic problems with known ground-truth and real-world applications demonstrate the advantages of our method.
  arXiv  Detail & Related papers  (2024-08-18T06:27:30Z)
• Synthetic Tabular Data Validation: A Divergence-Based Approach [8.062368743143388]
  Divergences quantify discrepancies between data distributions. Traditional approaches calculate divergences independently for each feature. We propose a novel approach that uses divergence estimation to overcome the limitations of marginal comparisons.
  arXiv  Detail & Related papers  (2024-05-13T15:07:52Z)
• Scalable Bayesian inference for the generalized linear mixed model [2.45365913654612]
  We introduce a statistical inference algorithm at the intersection of AI and Bayesian inference. Our algorithm is an extension of gradient MCMC with novel contributions that address the treatment of correlated data. We apply our algorithm to a large electronic health records database.
  arXiv  Detail & Related papers  (2024-03-05T14:35:34Z)
• Fundamental Limits of Membership Inference Attacks on Machine Learning Models [29.367087890055995]
  Membership inference attacks (MIA) can reveal whether a particular data point was part of the training dataset, potentially exposing sensitive information about individuals. This article provides theoretical guarantees by exploring the fundamental statistical limitations associated with MIAs on machine learning models.
  arXiv  Detail & Related papers  (2023-10-20T19:32:54Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• On the Joint Interaction of Models, Data, and Features [82.60073661644435]
  We introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. Based on these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form.
  arXiv  Detail & Related papers  (2023-06-07T21:35:26Z)
• Excess risk analysis for epistemic uncertainty with application to variational inference [110.4676591819618]
  We present a novel EU analysis in the frequentist setting, where data is generated from an unknown distribution. We show a relation between the generalization ability and the widely used EU measurements, such as the variance and entropy of the predictive distribution. We propose new variational inference that directly controls the prediction and EU evaluation performances based on the PAC-Bayesian theory.
  arXiv  Detail & Related papers  (2022-06-02T12:12:24Z)
• A Unifying Framework for Some Directed Distances in Statistics [0.0]
  Density-based directed distances -- particularly known as divergences -- are widely used in statistics. We provide a general framework which covers in particular both the density-based and distribution-function-based divergence approaches. We deduce new concepts of dependence between random variables, as alternatives to the celebrated mutual information.
  arXiv  Detail & Related papers  (2022-03-02T04:24:13Z)
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories [61.3299263929289]
  Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer an amortized estimator for the likelihood-to-evidence ratio. We show that this approach can be formulated in terms of mutual information between model parameters and simulated data.
  arXiv  Detail & Related papers  (2021-06-03T12:59:16Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.