Training Normalizing Flows with the Precision-Recall Divergence
- URL: http://arxiv.org/abs/2302.00628v2
- Date: Thu, 2 Feb 2023 16:46:03 GMT
- Title: Training Normalizing Flows with the Precision-Recall Divergence
- Authors: Alexandre Verine, Benjamin Negrevergne, Muni Sreenivas Pydi, Yann
Chevaleyre
- Abstract summary: We show that achieving a specified precision-recall trade-off corresponds to minimising -divergences from a family we call the em PR-divergences
We propose a novel generative model that is able to train a normalizing flow to minimise any -divergence, and in particular, achieve a given precision-recall trade-off.
- Score: 73.92251251511199
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Generative models can have distinct mode of failures like mode dropping and
low quality samples, which cannot be captured by a single scalar metric. To
address this, recent works propose evaluating generative models using precision
and recall, where precision measures quality of samples and recall measures the
coverage of the target distribution. Although a variety of discrepancy measures
between the target and estimated distribution are used to train generative
models, it is unclear what precision-recall trade-offs are achieved by various
choices of the discrepancy measures. In this paper, we show that achieving a
specified precision-recall trade-off corresponds to minimising -divergences
from a family we call the {\em PR-divergences }. Conversely, any -divergence
can be written as a linear combination of PR-divergences and therefore
correspond to minimising a weighted precision-recall trade-off. Further, we
propose a novel generative model that is able to train a normalizing flow to
minimise any -divergence, and in particular, achieve a given precision-recall
trade-off.
Related papers
- DistPred: A Distribution-Free Probabilistic Inference Method for Regression and Forecasting [14.390842560217743]
We propose a novel approach called DistPred for regression and forecasting tasks.
We transform proper scoring rules that measure the discrepancy between the predicted distribution and the target distribution into a differentiable discrete form.
This allows the model to sample numerous samples in a single forward pass to estimate the potential distribution of the response variable.
arXiv Detail & Related papers (2024-06-17T10:33:00Z) - Rejection via Learning Density Ratios [50.91522897152437]
Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.
We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.
Our framework is tested empirically over clean and noisy datasets.
arXiv Detail & Related papers (2024-05-29T01:32:17Z) - Few-Shot Recalibration of Language Models [23.829795148520834]
We train a recalibration model that takes in a few unlabeled examples from any given slice and predicts a curve that remaps confidence scores to be more accurate for that slice.
Our trained model can recalibrate for arbitrary new slices, without using any labeled data from that slice.
Experiments show that our few-shot recalibrator consistently outperforms existing calibration methods.
arXiv Detail & Related papers (2024-03-27T06:25:40Z) - Accuracy-Preserving Calibration via Statistical Modeling on Probability
Simplex [5.64979077798699]
We propose an accuracy-preserving calibration method using the Concrete distribution as the probabilistic model on the probability simplex.
We demonstrate that the proposed method can outperform previous methods in accuracy-preserving calibration tasks using benchmarks.
arXiv Detail & Related papers (2024-02-21T12:39:20Z) - Calibration by Distribution Matching: Trainable Kernel Calibration
Metrics [56.629245030893685]
We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression.
These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization.
We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
arXiv Detail & Related papers (2023-10-31T06:19:40Z) - Precision-Recall Divergence Optimization for Generative Modeling with
GANs and Normalizing Flows [54.050498411883495]
We develop a novel training method for generative models, such as Generative Adversarial Networks and Normalizing Flows.
We show that achieving a specified precision-recall trade-off corresponds to minimizing a unique $f$-divergence from a family we call the textitPR-divergences.
Our approach improves the performance of existing state-of-the-art models like BigGAN in terms of either precision or recall when tested on datasets such as ImageNet.
arXiv Detail & Related papers (2023-05-30T10:07:17Z) - On Calibrating Diffusion Probabilistic Models [78.75538484265292]
diffusion probabilistic models (DPMs) have achieved promising results in diverse generative tasks.
We propose a simple way for calibrating an arbitrary pretrained DPM, with which the score matching loss can be reduced and the lower bounds of model likelihood can be increased.
Our calibration method is performed only once and the resulting models can be used repeatedly for sampling.
arXiv Detail & Related papers (2023-02-21T14:14:40Z) - Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios.
We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
arXiv Detail & Related papers (2022-04-13T12:43:12Z) - Achieving Efficiency in Black Box Simulation of Distribution Tails with
Self-structuring Importance Samplers [1.6114012813668934]
The paper presents a novel Importance Sampling (IS) scheme for estimating distribution of performance measures modeled with a rich set of tools such as linear programs, integer linear programs, piecewise linear/quadratic objectives, feature maps specified with deep neural networks, etc.
arXiv Detail & Related papers (2021-02-14T03:37:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.