A Theoretical Framework for Statistical Evaluability of Generative Models

• URL: http://arxiv.org/abs/2604.05324v1
• Date: Tue, 07 Apr 2026 01:53:59 GMT
• Title: A Theoretical Framework for Statistical Evaluability of Generative Models
• Abstract summary: We introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics.<n>We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs) and Rényi divergences.<n>We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors.<n>In contrast, Rényi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events.
• Score: 57.9316356505791
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Statistical evaluation aims to estimate the generalization performance of a model using held-out i.i.d.\ test data sampled from the ground-truth distribution. In supervised learning settings such as classification, performance metrics such as error rate are well-defined, and test error reliably approximates population error given sufficiently large datasets. In contrast, evaluation is more challenging for generative models due to their open-ended nature: it is unclear which metrics are appropriate and whether such metrics can be reliably evaluated from finite samples. In this work, we introduce a theoretical framework for evaluating generative models and establish evaluability results for commonly used metrics. We study two categories of metrics: test-based metrics, including integral probability metrics (IPMs), and Rényi divergences. We show that IPMs with respect to any bounded test class can be evaluated from finite samples up to multiplicative and additive approximation errors. Moreover, when the test class has finite fat-shattering dimension, IPMs can be evaluated with arbitrary precision. In contrast, Rényi and KL divergences are not evaluable from finite samples, as their values can be critically determined by rare events. We also analyze the potential and limitations of perplexity as an evaluation method.

Related papers

• What Do Learned Models Measure? [0.0]
  In many scientific and data-driven applications, machine learning models are increasingly used as measurement instruments.<n>We show that standard evaluation criteria in machine learning, including generalization error, calibration, and robustness, do not guarantee measurement stability.
  arXiv  Detail & Related papers  (2026-01-26T09:00:48Z)
• Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check [60.77691669644931]
  We propose Functional Alignment for Distributional Equivalence (FADE), a novel metric that measures distributional similarity between unlearned and reference models.<n>We show that FADE captures functional alignment across the entire output distribution, providing a principled assessment of genuine unlearning.<n>These findings expose fundamental gaps in current evaluation practices and demonstrate that FADE provides a more robust foundation for developing and assessing truly effective unlearning methods.
  arXiv  Detail & Related papers  (2025-10-14T20:50:30Z)
• Lower Bounds on the MMSE of Adversarially Inferring Sensitive Features [4.8998185508205765]
  We propose an adversarial evaluation framework for sensitive feature inference based on a finite sample size and linear predictive models.<n>Our approach establishes theoretical lower bounds on the true MMSE of inferring sensitive features from noisy observations of other correlated features.
  arXiv  Detail & Related papers  (2025-05-13T22:39:24Z)
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models [48.96686419141881]
  We introduce the first formal probabilistic evaluation framework for Large Language Models (LLMs)<n> Namely, we propose novel metrics with high probability guarantees concerning the output distribution of a model.<n>Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment.
  arXiv  Detail & Related papers  (2024-10-04T15:44:23Z)
• Faithful Model Evaluation for Model-Based Metrics [22.753929098534403]
  We establish the mathematical foundation of significance testing for model-based metrics. We show that considering metric model errors to calculate sample variances for model-based metrics changes the conclusions in certain experiments.
  arXiv  Detail & Related papers  (2023-12-19T19:41:33Z)
• Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples [25.657798631897908]
  Feature Likelihood Divergence provides a comprehensive trichotomic evaluation of generative models. We empirically demonstrate the ability of FLD to identify overfitting problem cases, even when previously proposed metrics fail.
  arXiv  Detail & Related papers  (2023-02-09T04:57:27Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)
• Density of States Estimation for Out-of-Distribution Detection [69.90130863160384]
  DoSE is the density of states estimator. We demonstrate DoSE's state-of-the-art performance against other unsupervised OOD detectors.
  arXiv  Detail & Related papers  (2020-06-16T16:06:25Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.