Revisiting Reweighted Risk for Calibration: AURC, Focal, and Inverse Focal Loss

• URL: http://arxiv.org/abs/2505.23463v4
• Date: Thu, 09 Oct 2025 09:25:31 GMT
• Title: Revisiting Reweighted Risk for Calibration: AURC, Focal, and Inverse Focal Loss
• Authors: Han Zhou, Sebastian G. Gruber, Teodora Popordanoska, Matthew B. Blaschko,
• Abstract summary: We revisit a class of weighted risk functions commonly used in deep learning and establish a principled connection between calibration error and selective classification.<n>We show that minimizing calibration error is closely linked to the selective classification paradigm and demonstrate that optimizing selective risk in low-confidence region naturally leads to improved calibration.
• Score: 24.141034387131267
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Several variants of reweighted risk functionals, such as focal loss, inverse focal loss, and the Area Under the Risk--Coverage Curve (AURC), have been proposed for improving model calibration, yet their theoretical connections to calibration errors remain unclear. In this paper, we revisit a broad class of weighted risk functions commonly used in deep learning and establish a principled connection between calibration error and selective classification. We show that minimizing calibration error is closely linked to the selective classification paradigm and demonstrate that optimizing selective risk in low-confidence region naturally leads to improved calibration. This loss shares a similar reweighting strategy with dual focal loss but offers greater flexibility through the choice of confidence score functions (CSFs). Our approach uses a bin-based cumulative distribution function (CDF) approximation, enabling efficient gradient-based optimization without requiring expensive sorting and achieving $O(nK)$ complexity. Empirical evaluations demonstrate that our method achieves competitive calibration performance across a range of datasets and model architectures.

Related papers

• SGIC: A Self-Guided Iterative Calibration Framework for RAG [45.17496149653415]
  Large language models (LLMs) capitalize on their robust in-context reasoning.<n>We present a new framework that employs uncertainty scores as a tool.<n>We also introduce an innovative approach for constructing an iterative self-calibration training set.
  arXiv  Detail & Related papers  (2025-06-19T09:45:13Z)
• Know What You Don't Know: Uncertainty Calibration of Process Reward Models [8.958124143194512]
  Even state-of-the-art PRMs can be poorly calibrated and often overestimate success probabilities.<n>We present a calibration approach, performed via quantile regression, that PRM outputs to better align with true success probabilities.
  arXiv  Detail & Related papers  (2025-06-11T02:39:26Z)
• CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment [7.702016079410588]
  We introduce CLUE (Calibration via Learning Uncertainty-Error Alignment), a novel approach that aligns predicted uncertainty with observed error during training.<n>We show that CLUE achieves superior calibration quality and competitive predictive performance with respect to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-05-28T19:23:47Z)
• Uniform convergence of the smooth calibration error and its relationship with functional gradient [15.875913304310297]
  This work focuses on the smooth calibration error (CE) and provides a uniform convergence bound.<n>We analyze three representative algorithms: gradient boosting trees, kernel boosting, and two-layer neural networks.<n>Our results offer new theoretical insights and practical guidance for designing reliable probabilistic models.
  arXiv  Detail & Related papers  (2025-05-26T01:23:56Z)
• Calibration Strategies for Robust Causal Estimation: Theoretical and Empirical Insights on Propensity Score-Based Estimators [0.6562256987706128]
  partitioning of data for estimation and calibration critically impacts the performance of propensity score based estimators.<n>We extend recent advances in calibration techniques for propensity score estimation, improving the robustness of propensity scores in challenging settings.
  arXiv  Detail & Related papers  (2025-03-21T16:41:10Z)
• Exploring the Potential of Bilevel Optimization for Calibrating Neural Networks [0.7499722271664147]
  Modern neural networks are poorly calibrated, resulting in predicted confidence scores that are difficult to use.<n>This article explores improving confidence estimation and calibration through the application of bilevel optimization.<n>A self-calibrating bilevel neural-network training approach is introduced to improve a model's predicted confidence scores.
  arXiv  Detail & Related papers  (2025-03-17T12:34:55Z)
• Calibrating Deep Neural Network using Euclidean Distance [5.3612053942581275]
  In machine learning, Focal Loss is commonly used to reduce misclassification rates by emphasizing hard-to-classify samples.<n>High calibration error indicates a misalignment between predicted probabilities and actual outcomes, affecting model reliability.<n>This research introduces a novel loss function called Focal Loss (FCL), designed to improve probability calibration while retaining the advantages of Focal Loss in handling difficult samples.
  arXiv  Detail & Related papers  (2024-10-23T23:06:50Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Prior Constraints-based Reward Model Training for Aligning Large Language Models [58.33118716810208]
  This paper proposes a Prior Constraints-based Reward Model (namely PCRM) training method to mitigate this problem. PCRM incorporates prior constraints, specifically, length ratio and cosine similarity between outputs of each comparison pair, during reward model training to regulate optimization magnitude and control score margins. Experimental results demonstrate that PCRM significantly improves alignment performance by effectively constraining reward score scaling.
  arXiv  Detail & Related papers  (2024-04-01T07:49:11Z)
• Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization [59.758009422067]
  We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. We propose a new uncertainty Bellman equation (UBE) whose solution converges to the true posterior variance over values. We introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC) that can be applied for either risk-seeking or risk-averse policy optimization.
  arXiv  Detail & Related papers  (2023-12-07T15:55:58Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback [57.6775169085215]
  Risk-sensitive reinforcement learning aims to optimize policies that balance the expected reward and risk. We present a novel framework that employs an Iterated Conditional Value-at-Risk (CVaR) objective under both linear and general function approximations. We propose provably sample-efficient algorithms for this Iterated CVaR RL and provide rigorous theoretical analysis.
  arXiv  Detail & Related papers  (2023-07-06T08:14:54Z)
• Modular Conformal Calibration [80.33410096908872]
  We introduce a versatile class of algorithms for recalibration in regression. This framework allows one to transform any regression model into a calibrated probabilistic model. We conduct an empirical study of MCC on 17 regression datasets.
  arXiv  Detail & Related papers  (2022-06-23T03:25:23Z)
• Error-based Knockoffs Inference for Controlled Feature Selection [49.99321384855201]
  We propose an error-based knockoff inference method by integrating the knockoff features, the error-based feature importance statistics, and the stepdown procedure together. The proposed inference procedure does not require specifying a regression model and can handle feature selection with theoretical guarantees.
  arXiv  Detail & Related papers  (2022-03-09T01:55:59Z)
• Learning convex regularizers satisfying the variational source condition for inverse problems [4.2917182054051]
  We propose adversarial convex regularization (ACR) to learn data-driven convex regularizers via adversarial training. We leverage the variational source condition (SC) during training to enforce that the ground-truth images minimize the variational loss corresponding to the learned convex regularizer. The resulting regularizer (ACR-SC) performs on par with the ACR, but unlike ACR, comes with a quantitative convergence rate estimate.
  arXiv  Detail & Related papers  (2021-10-24T20:05:59Z)
• Federated Distributionally Robust Optimization for Phase Configuration of RISs [106.4688072667105]
  We study the problem of robust reconfigurable intelligent surface (RIS)-aided downlink communication over heterogeneous RIS types in a supervised learning setting. By modeling downlink communication over heterogeneous RIS designs as different workers that learn how to optimize phase configurations in a distributed manner, we solve this distributed learning problem. Our proposed algorithm requires fewer communication rounds to achieve the same worst-case distribution test accuracy compared to competitive baselines.
  arXiv  Detail & Related papers  (2021-08-20T07:07:45Z)
• Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence [66.83161885378192]
  Area under ROC (AUROC) and precision-recall curves (AUPRC) are common metrics for evaluating classification performance for imbalanced problems. We propose a technical method to optimize AUPRC for deep learning.
  arXiv  Detail & Related papers  (2021-04-18T06:22:21Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.