Emergent Asymmetry of Precision and Recall for Measuring Fidelity and Diversity of Generative Models in High Dimensions

• URL: http://arxiv.org/abs/2306.09618v2
• Date: Tue, 18 Jul 2023 20:39:43 GMT
• Title: Emergent Asymmetry of Precision and Recall for Measuring Fidelity and Diversity of Generative Models in High Dimensions
• Authors: Mahyar Khayatkhoei, Wael AbdAlmageed
• Abstract summary: We show that as the number of dimensions grows, two model distributions can have vastly different Precision and Recall regardless of their respective distributions. We then provide simple yet effective modifications to these metrics to construct symmetric metrics regardless of the number of dimensions.
• Score: 15.777510451215752
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Precision and Recall are two prominent metrics of generative performance, which were proposed to separately measure the fidelity and diversity of generative models. Given their central role in comparing and improving generative models, understanding their limitations are crucially important. To that end, in this work, we identify a critical flaw in the common approximation of these metrics using k-nearest-neighbors, namely, that the very interpretations of fidelity and diversity that are assigned to Precision and Recall can fail in high dimensions, resulting in very misleading conclusions. Specifically, we empirically and theoretically show that as the number of dimensions grows, two model distributions with supports at equal point-wise distance from the support of the real distribution, can have vastly different Precision and Recall regardless of their respective distributions, hence an emergent asymmetry in high dimensions. Based on our theoretical insights, we then provide simple yet effective modifications to these metrics to construct symmetric metrics regardless of the number of dimensions. Finally, we provide experiments on real-world datasets to illustrate that the identified flaw is not merely a pathological case, and that our proposed metrics are effective in alleviating its impact.

Related papers

• A Unifying Information-theoretic Perspective on Evaluating Generative Models [5.524685807042777]
  Several recent approaches utilize "precision" and "recall," borrowed from the classification domain, to individually quantify the output fidelity (realism) and output diversity (representation of the real data variation) We unify a class of kth-nearest-neighbors (kNN)-based metrics under an information-theoretic lens using approaches from kNN density estimation. We propose a tri-dimensional metric composed of Precision Cross-Entropy (PCE), Recall Cross-Entropy (RCE), and Recall Entropy (RE)
  arXiv  Detail & Related papers  (2024-12-18T21:17:02Z)
• Measuring Orthogonality in Representations of Generative Models [81.13466637365553]
  In unsupervised representation learning, models aim to distill essential features from high-dimensional data into lower-dimensional learned representations. Disentanglement of independent generative processes has long been credited with producing high-quality representations. We propose two novel metrics: Importance-Weighted Orthogonality (IWO) and Importance-Weighted Rank (IWR)
  arXiv  Detail & Related papers  (2024-07-04T08:21:54Z)
• Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models [7.770029179741429]
  We propose P-precision and P-recall (PP&PR), based on a probabilistic approach that address the problems. We show that our PP&PR provide more reliable estimates for comparing fidelity and diversity than the existing metrics.
  arXiv  Detail & Related papers  (2023-09-04T13:19:17Z)
• Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation [50.920911532133154]
  The intrinsic ill-posedness and ordinal-sensitive nature of monocular depth estimation (MDE) models pose major challenges to the estimation of uncertainty degree. We propose to model the uncertainty of MDE models from the perspective of the inherent probability distributions. By simply introducing additional training regularization terms, our model, with surprisingly simple formations and without requiring extra modules or multiple inferences, can provide uncertainty estimations with state-of-the-art reliability.
  arXiv  Detail & Related papers  (2023-07-19T12:11:15Z)
• Implicit Bayes Adaptation: A Collaborative Transport Approach [25.96406219707398]
  We show that domain adaptation is rooted in the intrinsic representations of the respective data, which are inherently lying in a non-linear submanifold embedded in a higher dimensional Euclidean space. We show that this is tantamount to an implicit Bayesian framework, which we demonstrate to be viable for a more robust and better-performing approach to domain adaptation.
  arXiv  Detail & Related papers  (2023-04-17T14:13:40Z)
• Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes [52.92110730286403]
  It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension. We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
  arXiv  Detail & Related papers  (2022-10-14T08:09:33Z)
• Structure and Distribution Metric for Quantifying the Quality of Uncertainty: Assessing Gaussian Processes, Deep Neural Nets, and Deep Neural Operators for Regression [0.0]
  We propose two comparison metrics that may be implemented to arbitrary dimensions in regression tasks. The structure metric assesses the similarity in shape and location of uncertainty with the true error, while the distribution metric quantifies the supported magnitudes between the two. We apply these metrics to Gaussian Processes (GPs), Ensemble Deep Neural Nets (DNNs), and Ensemble Deep Neural Operators (DNOs) on high-dimensional and nonlinear test cases.
  arXiv  Detail & Related papers  (2022-03-09T04:16:31Z)
• Evaluating Sensitivity to the Stick-Breaking Prior in Bayesian Nonparametrics [85.31247588089686]
  We show that variational Bayesian methods can yield sensitivities with respect to parametric and nonparametric aspects of Bayesian models. We provide both theoretical and empirical support for our variational approach to Bayesian sensitivity analysis.
  arXiv  Detail & Related papers  (2021-07-08T03:40:18Z)
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics [61.49826776409194]
  We analyze a corpus of models made publicly-available for a contest to predict the generalization accuracy of neural network (NN) models. We identify what amounts to a Simpson's paradox: where "scale" metrics perform well overall but perform poorly on sub partitions of the data. We present two novel shape metrics, one data-independent, and the other data-dependent, which can predict trends in the test accuracy of a series of NNs.
  arXiv  Detail & Related papers  (2021-06-01T19:19:49Z)
• Rethink Maximum Mean Discrepancy for Domain Adaptation [77.2560592127872]
  This paper theoretically proves two essential facts: 1) minimizing the Maximum Mean Discrepancy equals to maximize the source and target intra-class distances respectively but jointly minimize their variance with some implicit weights, so that the feature discriminability degrades. Experiments on several benchmark datasets not only prove the validity of theoretical results but also demonstrate that our approach could perform better than the comparative state-of-art methods substantially.
  arXiv  Detail & Related papers  (2020-07-01T18:25:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.