On Calibration in Multi-Distribution Learning

• URL: http://arxiv.org/abs/2412.14142v1
• Date: Wed, 18 Dec 2024 18:41:40 GMT
• Title: On Calibration in Multi-Distribution Learning
• Authors: Rajeev Verma, Volker Fischer, Eric Nalisnick,
• Abstract summary: We study the calibration properties of multi-distribution learning (MDL) We first derive the Bayes optimal rule for MDL, demonstrating that it maximizes the generalized entropy of the associated loss function. Our analysis reveals that while this approach ensures minimal worst-case loss, it can lead to non-uniform calibration errors across the multiple distributions.
• Score: 6.184670046923719
• License:
• Abstract: Modern challenges of robustness, fairness, and decision-making in machine learning have led to the formulation of multi-distribution learning (MDL) frameworks in which a predictor is optimized across multiple distributions. We study the calibration properties of MDL to better understand how the predictor performs uniformly across the multiple distributions. Through classical results on decomposing proper scoring losses, we first derive the Bayes optimal rule for MDL, demonstrating that it maximizes the generalized entropy of the associated loss function. Our analysis reveals that while this approach ensures minimal worst-case loss, it can lead to non-uniform calibration errors across the multiple distributions and there is an inherent calibration-refinement trade-off, even at Bayes optimality. Our results highlight a critical limitation: despite the promise of MDL, one must use caution when designing predictors tailored to multiple distributions so as to minimize disparity.

Related papers

• Rethinking Early Stopping: Refine, Then Calibrate [49.966899634962374]
  We show that calibration error and refinement error are not minimized simultaneously during training. We introduce a new metric for early stopping and hyper parameter tuning that makes it possible to minimize refinement error during training. Our method integrates seamlessly with any architecture and consistently improves performance across diverse classification tasks.
  arXiv  Detail & Related papers  (2025-01-31T15:03:54Z)
• Quantifying the Prediction Uncertainty of Machine Learning Models for Individual Data [2.1248439796866228]
  This study investigates pNML's learnability for linear regression and neural networks. It demonstrates that pNML can improve the performance and robustness of these models on various tasks.
  arXiv  Detail & Related papers  (2024-12-10T13:58:19Z)
• Calibrated Multivariate Regression with Localized PIT Mappings [4.277516034244117]
  This paper introduces a novel post-hoc recalibration approach that addresses multivariate calibration for potentially misspecified models. We present two versions of our approach: one uses K-nearest neighbors, and the other uses normalizing flows. We demonstrate the effectiveness of our approach on two real data applications: recalibrating a deep neural network's currency exchange rate forecast and improving a regression model for childhood malnutrition in India.
  arXiv  Detail & Related papers  (2024-09-17T02:41:03Z)
• Calibrating Large Language Models with Sample Consistency [76.23956851098598]
  We explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. Results show that consistency-based calibration methods outperform existing post-hoc approaches. We offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.
  arXiv  Detail & Related papers  (2024-02-21T16:15:20Z)
• Distributional Robustness Bounds Generalization Errors [2.3940819037450987]
  We suggest a quantitative definition for "distributional robustness" We show that Bayesian methods are distributionally robust in the probably approximately correct (PAC) sense. We show that generalization errors of machine learning models can be characterized using the distributional uncertainty of the nominal distribution.
  arXiv  Detail & Related papers  (2022-12-20T02:30:13Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• A Unified Framework for Multi-distribution Density Ratio Estimation [101.67420298343512]
  Binary density ratio estimation (DRE) provides the foundation for many state-of-the-art machine learning algorithms. We develop a general framework from the perspective of Bregman minimization divergence. We show that our framework leads to methods that strictly generalize their counterparts in binary DRE.
  arXiv  Detail & Related papers  (2021-12-07T01:23:20Z)
• Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions [91.63716984911278]
  We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
  arXiv  Detail & Related papers  (2021-11-11T14:28:12Z)
• Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation [99.92568326314667]
  We propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation. Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principle. We demonstrate that ACNML compares favorably to a number of prior techniques for uncertainty estimation in terms of calibration on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-11-05T08:04:34Z)
• Robustified Multivariate Regression and Classification Using Distributionally Robust Optimization under the Wasserstein Metric [11.383869751239166]
  We develop Distributionally Robust Optimization (DRO) formulations for Multivariate Linear Regression (MLR) and Multiclass Logistic Regression (MLG) We relax the DRO formulation into a regularized learning problem whose regularizer is a norm of the coefficient matrix. Experimental results show that our approach improves the predictive error by 7% -- 37% for MLR, and a metric of robustness by 100% for MLG.
  arXiv  Detail & Related papers  (2020-06-10T22:16:50Z)
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization [13.97417198693205]
  We show that our BNNs achieve higher accuracy on multiple benchmarks, including several image classification tasks. We also provide a new formulation for estimating the uncertainty on a given prediction, showing it performs in a more robust fashion against adversarial attacks.
  arXiv  Detail & Related papers  (2020-03-02T14:54:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.