Technical note on calibrating vision-language models under covariate shift

• URL: http://arxiv.org/abs/2502.07847v1
• Date: Tue, 11 Feb 2025 10:10:15 GMT
• Title: Technical note on calibrating vision-language models under covariate shift
• Authors: Behraj Khan, Rizwan Qureshi, Tahir Syed,
• Abstract summary: vision-language foundation models for low-shot vision classification have a limited ability to generalize to the target data distribution. We propose textitConfidence-Calibrated Covariate Shift Correction ($C3SC$), a unified framework to mitigate both covariate shift and confidence misalignment. $C3SC$ significantly improves robustness in calibration (ECE) by $5.82%$ at maximum.
• Score: 2.8470354623829577
• License:
• Abstract: Despite being a successful example of emerging capability, vision-language foundation models for low-shot vision classification have a limited ability to sufficiently generalize to the target data distribution due to sample poverty, leading to sensitivity to variations in the data. A popular mitigation strategy is finetuning over multiple datasets, but domain generalization is expensive when practiced in this manner. This work examines both covariate shift between pre-training data and the underspecified target data, and \textit{confidence misalignment}, where the model's prediction confidence amplified by the limited data availability. We propose \textit{Confidence-Calibrated Covariate Shift Correction ($C3SC$)}, a unified framework to mitigate both covariate shift and confidence misalignment. $C3SC$ leverages Fisher information penalty for covariate shift correction and confidence misalignment penalty (CMP) to lower confidence on misclassified examples. Experimental results across various vision and covariate shift datasets demonstrates that $C3SC$ significantly improves in calibration (ECE) by $5.82\%$ at maximum. $C3SC$ shows better robustness as well by showing $3.5\%$ improvement in accuracy metric on challenging covariate shift datasets, making $C3SC$ a promising solution for reliable real-world vision-language low-shot applications under distribution shift.

Related papers

• Technical report on label-informed logit redistribution for better domain generalization in low-shot classification with foundation models [0.0]
  Confidence calibration is an emerging challenge in real-world decision systems based on foundations models. We propose a penalty incorporated into loss objective that penalizes incorrect classifications whenever one is made during finetuning. We refer to it as textitconfidence misalignment penalty (CMP).
  arXiv  Detail & Related papers  (2025-01-29T11:54:37Z)
• SGCCNet: Single-Stage 3D Object Detector With Saliency-Guided Data Augmentation and Confidence Correction Mechanism [7.631190617438259]
  Single-stage point-based 3D object detectors face challenges such as inadequate learning of low-quality objects (ILQ) and misalignment between localization accuracy and classification confidence (MLC) For ILQ, SGCCNet adopts a Saliency-Guided Data Augmentation (SGDA) strategy to enhance the robustness of the model on low-quality objects. For MLC, we design a Confidence Correction Mechanism ( CCM) specifically for point-based multi-class detectors.
  arXiv  Detail & Related papers  (2024-07-01T12:36:01Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)
• Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shift [50.98086766507025]
  We propose a simple-yet-effective data augmentation strategy, Adversarial Invariant Augmentation (AIA) AIA aims to extrapolate and generate new environments, while concurrently preserving the original stable features during the augmentation process.
  arXiv  Detail & Related papers  (2022-11-05T07:55:55Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Test-time Batch Statistics Calibration for Covariate Shift [66.7044675981449]
  We propose to adapt the deep models to the novel environment during inference. We present a general formulation $alpha$-BN to calibrate the batch statistics. We also present a novel loss function to form a unified test time adaptation framework Core.
  arXiv  Detail & Related papers  (2021-10-06T08:45:03Z)
• Closer Look at the Uncertainty Estimation in Semantic Segmentation under Distributional Shift [2.05617385614792]
  Uncertainty estimation for the task of semantic segmentation is evaluated under a varying level of domain shift. It was shown that simple color transformations already provide a strong baseline. ensemble of models was utilized in the self-training setting to improve the pseudo-labels generation.
  arXiv  Detail & Related papers  (2021-05-31T19:50:43Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.