Calibrating sufficiently
- URL: http://arxiv.org/abs/2105.07283v2
- Date: Tue, 18 May 2021 18:36:00 GMT
- Title: Calibrating sufficiently
- Authors: Dirk Tasche
- Abstract summary: Grouping loss refers to the gap between observable information and information actually exploited in the calibration exercise.
We investigate the relation between grouping loss and the concept of sufficiency, identifying comonotonicity as a useful criterion for sufficiency.
- Score: 2.1320960069210475
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: When probabilistic classifiers are trained and calibrated, the so-called
grouping loss component of the calibration loss can easily be overlooked.
Grouping loss refers to the gap between observable information and information
actually exploited in the calibration exercise. We investigate the relation
between grouping loss and the concept of sufficiency, identifying
comonotonicity as a useful criterion for sufficiency. We revisit the probing
reduction approach of Langford & Zadrozny (2005) and find that it produces an
estimator of probabilistic classifiers that reduces grouping loss. Finally, we
discuss Brier curves as tools to support training and 'sufficient' calibration
of probabilistic classifiers.
Related papers
- Calibration by Distribution Matching: Trainable Kernel Calibration
Metrics [56.629245030893685]
We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression.
These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization.
We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
arXiv Detail & Related papers (2023-10-31T06:19:40Z) - Scaling of Class-wise Training Losses for Post-hoc Calibration [6.0632746602205865]
We propose a new calibration method to synchronize the class-wise training losses.
We design a new training loss to alleviate the variance of class-wise training losses by using multiple class-wise scaling factors.
We validate the proposed framework by employing it in the various post-hoc calibration methods.
arXiv Detail & Related papers (2023-06-19T14:59:37Z) - Beyond calibration: estimating the grouping loss of modern neural
networks [68.8204255655161]
Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss.
We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings.
arXiv Detail & Related papers (2022-10-28T07:04:20Z) - Shift Happens: Adjusting Classifiers [2.8682942808330703]
Minimizing expected loss measured by a proper scoring rule, such as Brier score or log-loss (cross-entropy), is a common objective while training a probabilistic classifier.
We propose methods that transform all predictions to (re)equalize the average prediction and the class distribution.
We demonstrate experimentally that, when in practice the class distribution is known only approximately, there is often still a reduction in loss depending on the amount of shift and the precision to which the class distribution is known.
arXiv Detail & Related papers (2021-11-03T21:27:27Z) - Lower-bounded proper losses for weakly supervised classification [73.974163801142]
We discuss the problem of weakly supervised learning of classification, in which instances are given weak labels.
We derive a representation theorem for proper losses in supervised learning, which dualizes the Savage representation.
We experimentally demonstrate the effectiveness of our proposed approach, as compared to improper or unbounded losses.
arXiv Detail & Related papers (2021-03-04T08:47:07Z) - Label-Imbalanced and Group-Sensitive Classification under
Overparameterization [32.923780772605596]
Label-imbalanced and group-sensitive classification seeks to appropriately modify standard training algorithms to optimize relevant metrics.
We show that a logit-adjusted loss modification to standard empirical risk minimization might be ineffective in general.
We show that our results extend naturally to binary classification with sensitive groups, thus treating the two common types of imbalances (label/group) in a unifying way.
arXiv Detail & Related papers (2021-03-02T08:09:43Z) - On Focal Loss for Class-Posterior Probability Estimation: A Theoretical
Perspective [83.19406301934245]
We first prove that the focal loss is classification-calibrated, i.e., its minimizer surely yields the Bayes-optimal classifier.
We then prove that the focal loss is not strictly proper, i.e., the confidence score of the classifier does not match the true class-posterior probability.
Our proposed transformation significantly improves the accuracy of class-posterior probability estimation.
arXiv Detail & Related papers (2020-11-18T09:36:52Z) - Unsupervised Calibration under Covariate Shift [92.02278658443166]
We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it.
We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
arXiv Detail & Related papers (2020-06-29T21:50:07Z) - Calibrated Surrogate Losses for Adversarially Robust Classification [92.37268323142307]
We show that no convex surrogate loss is respect with respect to adversarial 0-1 loss when restricted to linear models.
We also show that if the underlying distribution satisfies the Massart's noise condition, convex losses can also be calibrated in the adversarial setting.
arXiv Detail & Related papers (2020-05-28T02:40:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.