Unified Approach for Weakly Supervised Multicalibration

• URL: http://arxiv.org/abs/2605.09857v1
• Date: Mon, 11 May 2026 01:30:09 GMT
• Title: Unified Approach for Weakly Supervised Multicalibration
• Abstract summary: Multicalibration requires predicted scores to agree with label probabilities across rich families of subgroups and score-dependent tests.<n>Existing methods require clean input-label pairs for evaluation and post-processing.<n>We propose a unified framework for estimating and correcting multicalibration under weak supervision.
• Score: 6.557328947642346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multicalibration requires predicted scores to agree with label probabilities across rich families of subgroups and score-dependent tests, but existing methods require clean input-label pairs for evaluation and post-processing. This assumption fails in weakly supervised learning (WSL) regimes -- including positive-unlabeled, unlabeled-unlabeled, and positive-confidence learning -- where clean labels are costly or unavailable even though reliable uncertainty estimates may be crucial. We address this gap by developing estimators of multicalibration error and post-hoc correction methods for WSL settings in which clean input-label pairs are unavailable. We propose a unified framework for estimating and correcting multicalibration under weak supervision by combining contamination-matrix risk rewrites with witness-based calibration constraints, yielding corrected multicalibration moments with finite-sample guarantees. We further propose weak-label multicalibration boost (WLMC), a generic post-hoc recalibration algorithm under weak supervision. Finally, we conduct experiments across multiple weak-supervision settings to evaluate multicalibration behavior and offer empirical insight into uncertainty estimation under weak supervision.

Related papers

• Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs [9.993859591441238]
  We propose a framework that reconciles supervised few-shot adaptation with conformal uncertainty estimation under non-exchangeability.<n> AlignCP learns a reweighted calibration distribution that reduces the score-level discrepancy between the labeled support set and the unlabeled query set.
  arXiv  Detail & Related papers  (2026-09-09T15:32:01Z)
• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering [19.55210880950831]
  Expected measures whether a model's predicted confidence aligns with its empirical accuracy, and is central to the reliable deployment of large language models (LLMs)<n>We introduce Sem-ECE, a calibration evaluation framework for open-ended question answering (QA)<n>We study two estimators within framework: Sem$$-ECE, the same-sample self-consistency score, and Sem$$-ECE, a held-out variant that separates answer selection from confidence evaluation.
  arXiv  Detail & Related papers  (2026-05-08T19:53:49Z)
• LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs [61.06744611795341]
  Medical vision-language models (VLMs) are strong zero-shot recognizers for medical imaging.<n>We propose texttttextbfLATA (Laplacian-Assisted Transductive Adaptation), a textittraining- and label-free refinement.<n>texttttextbfLATA sharpens zero-shot predictions without compromising exchangeability.
  arXiv  Detail & Related papers  (2026-02-19T16:45:38Z)
• Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning [53.9713678229744]
  Multi-instance partial-label learning (MIPL) is a weakly supervised framework that addresses the challenges of inexact supervision in both instance and label spaces.<n>Existing MIPL approaches often suffer from poor calibration, undermining reliability.<n>We propose a plug-and-play calibratable disambiguation loss (CDL) that simultaneously improves classification accuracy and calibration performance.
  arXiv  Detail & Related papers  (2025-12-19T16:58:31Z)
• DiCaP: Distribution-Calibrated Pseudo-labeling for Semi-Supervised Multi-Label Learning [83.94574004953346]
  Semi-supervised multi-label learning aims to leverage unlabeled data to improve the model's performance.<n>Most existing methods assign equal weights to all pseudo-labels regardless of their quality.<n>We propose Distribution-Calibrated Pseudo-labeling (DiCaP), a correctness-aware framework that estimates posterior precision to calibrate pseudo-label weights.
  arXiv  Detail & Related papers  (2025-11-25T11:55:02Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Learning from Similarity-Confidence and Confidence-Difference [0.07646713951724009]
  We propose a novel Weakly Supervised Learning (WSL) framework that leverages complementary weak supervision signals from multiple perspectives.<n>Specifically, we introduce SconfConfDiff Classification, a method that integrates two distinct forms of weaklabels.<n>We prove that both estimators achieve optimal convergence rates with respect to estimation error bounds.
  arXiv  Detail & Related papers  (2025-08-07T07:42:59Z)
• A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPScore Quality Estimates [3.902360015414256]
  We propose a simple strategy for generating and calibrating distributions of CLIPScore values.<n>We calibrate CLIPScore values for task-specific control variables, tackling the limitations.<n>Our method effectively detects erroneous words, while providing formal guarantees aligned with desired risk levels.
  arXiv  Detail & Related papers  (2025-04-01T22:25:00Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)
• Fair admission risk prediction with proportional multicalibration [0.16249424686052708]
  Multicalibration constrains calibration error among flexibly-defined subpopulations. It is possible for a decision-maker to learn to trust or distrust model predictions for specific groups. We propose proportional multicalibration, a criteria that constrains the percent calibration error among groups and within prediction bins.
  arXiv  Detail & Related papers  (2022-09-29T08:15:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.