Not all Jensen-Shannon Divergence Estimators are Equal

• URL: http://arxiv.org/abs/2606.16411v1
• Date: Mon, 15 Jun 2026 08:49:29 GMT
• Title: Not all Jensen-Shannon Divergence Estimators are Equal
• Abstract summary: We show that different protocols can yield non-comparable values for the Jensen-Shannon divergence.<n>We show that empirical Jensen-Shannon divergence values are inherently protocol-dependent, making explicit specification of the estimation procedure necessary for meaningful comparison.
• Score: 38.594313322631216
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The Jensen-Shannon divergence is widely reported as a scalar measure of fidelity for synthetic tabular data. Yet, in practice, it is estimated from finite samples using protocols that are often underspecified. This creates a measurement problem. Although the population divergence is well defined, the empirical value depends on the estimator family, sampling protocol, calibration, dimensionality, and class balance. We show that different protocols can yield non-comparable values: marginal-based estimators ignore dependencies in the joint distribution and can severely underestimate divergence, while classifier-based estimators capture joint structure but exhibit strong estimator dependence. We systematically study this behavior across controlled settings with reference divergences and real-world synthetic tabular benchmarks. Our analysis reveals dependence blindness in marginal estimators, prior-shift bias under class imbalance, and estimator sensitivity in high dimensions. To address prior shift, we derive a closed-form posterior correction for classifier-based Jensen-Shannon estimation. Our results show that empirical Jensen-Shannon divergence values are inherently protocol-dependent, making explicit specification of the estimation procedure necessary for meaningful comparison. We provide practical guidelines and an open-source tool for estimator-aware Jensen-Shannon evaluation.

Related papers

• ARC: Adaptive Robust Joint State and Covariance Estimation [42.045109659898465]
  Sensor measurements are frequently corrupted by outliers and non-Gaussian noise.<n>These imperfections in the sensor data can cause classical state estimators to generate biased and unreliable state and uncertainty estimates.<n>This paper proposes a unified Block-Coordinate Descent framework that combines a norm-aware adaptive robust loss, and an Iteratively Reweighted Least-Squares state update.
  arXiv  Detail & Related papers  (2026-06-18T16:09:50Z)
• Beyond Consistency: Inference for the Relative risk functional in Deep Nonparametric Cox Models [7.2275111013288535]
  It is unclear how Cox-based optimization error propagates to population risk under partial likelihood.<n>It is also unclear how pointwise bias can be controlled, and how ensemble-based uncertainty behaves under realistic variance regimes.<n>We develop an distribution theory for deep Cox estimators that addresses these issues.
  arXiv  Detail & Related papers  (2026-03-25T01:43:12Z)
• Robust Score Matching [1.2835555561822447]
  We develop a robust score matching procedure that yields consistent parameter estimates in settings where the observed data has been contaminated.<n>A special appeal of the proposed method is that it retains convexity in exponential family models.<n>Support recovery is studied in numerical experiments and on a precipitation dataset.
  arXiv  Detail & Related papers  (2025-01-09T09:46:27Z)
• Label Shift Estimators for Non-Ignorable Missing Data [2.605549784939959]
  We consider the problem of estimating the mean of a random variable Y subject to non-ignorable missingness, i.e., where the missingness mechanism depends on Y. We use our approach to estimate disease prevalence using a large health survey, comparing ignorable and non-ignorable approaches.
  arXiv  Detail & Related papers  (2023-10-27T16:50:13Z)
• The Representation Jensen-Shannon Divergence [0.0]
  Quantifying the difference between probability distributions is crucial in machine learning. This work proposes the representation Jensen-Shannon divergence (RJSD), a novel measure inspired by the traditional Jensen-Shannon divergence. Our results demonstrate RJSD's superiority in two-sample testing, distribution shift detection, and unsupervised domain adaptation.
  arXiv  Detail & Related papers  (2023-05-25T19:44:36Z)
• Multi-Fidelity Covariance Estimation in the Log-Euclidean Geometry [0.0]
  We introduce a multi-fidelity estimator of covariance matrices that employs the log-Euclidean geometry of the symmetric positive-definite manifold. We develop an optimal sample allocation scheme that minimizes the mean-squared error of the estimator given a fixed budget. Evaluations of our approach using data from physical applications demonstrate more accurate metric learning and speedups of more than one order of magnitude compared to benchmarks.
  arXiv  Detail & Related papers  (2023-01-31T16:33:46Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)
• Nonparametric Score Estimators [49.42469547970041]
  Estimating the score from a set of samples generated by an unknown distribution is a fundamental task in inference and learning of probabilistic models. We provide a unifying view of these estimators under the framework of regularized nonparametric regression. We propose score estimators based on iterative regularization that enjoy computational benefits from curl-free kernels and fast convergence.
  arXiv  Detail & Related papers  (2020-05-20T15:01:03Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)
• Estimating Gradients for Discrete Random Variables by Sampling without Replacement [93.09326095997336]
  We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement. We show that our estimator can be derived as the Rao-Blackwellization of three different estimators.
  arXiv  Detail & Related papers  (2020-02-14T14:15:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.