Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift

• URL: http://arxiv.org/abs/2507.09222v2
• Date: Sun, 20 Jul 2025 07:22:45 GMT
• Title: Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift
• Abstract summary: Foundation models like CLIP and SAM have advanced computer vision and medical imaging via low-shot transfer learning, aiding CADD with limited data.<n>We propose textbfStaRFM, a fusion of Fisher information penalty (FIP) and confidence misalignment penalty (CMP) tackling both challenges.<n>It applies FIP, extended to 3D via patch-wise regularization, to reduce embedding shift, and CMP, reformulated for voxel-level predictions, to segmentation uncertainty.
• Score: 2.292525568003776
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Foundation models like CLIP and SAM have advanced computer vision and medical imaging via low-shot transfer learning, aiding CADD with limited data. However, their deployment faces two key challenges. \textit{distribution shift} where pre-training and post-training data distributions differ (e.g., due to inter-center image acquisition) and \textit{confidence misalignment}, which leads to overconfident errors. These issues surface differently, vision-language models (e.g., CLIP) suffer from 2D embedding shift (image-text misalignment), while medical models (e.g., SAM) encounter 3D domain shifts (e.g., scanner variation) and voxel-wise calibration need. Existing solutions are domain-specific. We propose \textbf{StaRFM}, a fusion of Fisher information penalty (FIP) and confidence misalignment penalty (CMP) tackling both challenges. It applies FIP, extended to 3D via patch-wise regularization, to reduce embedding shift, and CMP, reformulated for voxel-level predictions, to calibrate segmentation uncertainty. We derive PAC-Bayes bounds. FIP controls generalization via the Fisher-Rao norm, and CMP reduces calibration error via Brier score minimization. StaRFM surpasses baselines by \texttt{+}3.5\% accuracy and 28\% lower ECE on 19 vision datasets (e.g., ImageNet, Office-Home), achieves +4.2\% DSC over SAM-FT and 4.8mm HD95 on medical benchmarks (e.g., BraTS, ATLAS), and reduces cross-domain gaps by up to 20\%. The framework is plug-and-play, requiring minimal architectural changes. Code and models are available at: \href{https://anonymous.4open.science/r/StaRFM-C0CD/}{\textcolor{blue}{\underline{StaRFM}}}

Related papers

• DepthART: Scaling Foundation Monocular Depth to Tiny Models [51.10174763031444]
  DepthART is a compact MDE model for on-device deployment across diverse scenes.<n>We show how DepthART consistently surpasses previous tiny baselines in both zero-shot generalization and metric accuracy.<n>We also provide a scalable model family, with DepthART-S reaching 347/245 FPS (strict FP32) on an GTX A6000 at $2242/4482$, 102 FPS (TF32) on a Orin NX 8GB, and over 15 FPS (FP32) on a Jetson Nano 4GB.
  arXiv  Detail & Related papers  (2026-07-19T06:54:52Z)
• Large-Small Model Collaboration for Farmland Semantic Change Detection [10.406965352353307]
  Farmland Semantic Change Detection essential for cultivated land protection.<n>Existing benchmarks and models are insufficient for fine-grained farmland conversion monitoring.<n>We propose a large-small collaborative SCD framework that integrates a task-driven small visual model with a frozen large vision-language model.<n>Our method achieves 97.63% F1, 96.32% IoU, and 96.35% SCD_IoU_mean on HZNU-FCD with only 6.65M trainable parameters.
  arXiv  Detail & Related papers  (2026-05-12T15:40:19Z)
• I Can't Believe TTA Is Not Better: When Test-Time Augmentation Hurts Medical Image Classification [0.0]
  Test-time augmentation (TTA) is widely assumed to improve classification accuracy in medical imaging.<n>We present a systematic empirical study challenging this assumption across three MedMNIST v2 benchmarks and four architectures.<n>Our principal finding is that TTA with standard augmentation pipelines consistently degrades accuracy relative to single-pass inference.
  arXiv  Detail & Related papers  (2026-04-06T23:52:10Z)
• LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs [61.06744611795341]
  Medical vision-language models (VLMs) are strong zero-shot recognizers for medical imaging.<n>We propose texttttextbfLATA (Laplacian-Assisted Transductive Adaptation), a textittraining- and label-free refinement.<n>texttttextbfLATA sharpens zero-shot predictions without compromising exchangeability.
  arXiv  Detail & Related papers  (2026-02-19T16:45:38Z)
• A3-TTA: Adaptive Anchor Alignment Test-Time Adaptation for Image Segmentation [17.122762119608144]
  Test-Time Adaptation (TTA) offers a practical solution for deploying image segmentation models under domain shift.<n>We propose textbfA3-TTA, a framework that constructs reliable pseudo-labels through anchor-guided supervision.<n>A3-TTA significantly improves average Dice scores by 10.40 to 17.68 percentage points compared to the source model.
  arXiv  Detail & Related papers  (2026-02-03T09:18:11Z)
• Not Quite Anything: Overcoming SAMs Limitations for 3D Medical Imaging [0.0]
  Foundation segmentation models such as SAM and SAM-2 perform well on natural images but struggle with brain MRIs.<n>We propose a compositional alternative where the foundation model output is treated as an additional input channel.<n>We generate SAM-2 prompts by using a lightweight 3D U-Net that was previously trained on MRI segmentation.
  arXiv  Detail & Related papers  (2025-11-22T05:29:27Z)
• MuSc-V2: Zero-Shot Multimodal Industrial Anomaly Classification and Segmentation with Mutual Scoring of Unlabeled Samples [8.323617762880533]
  We propose a Mutual Scoring framework (MuSc-V2) for zero-shot anomaly classification (AC/AS)<n>Our framework flexibly works on both the full dataset and smaller subsets with consistently robust performance.<n>In aid of the novel framework, MuSc-V2 achieves significant performance improvements.
  arXiv  Detail & Related papers  (2025-11-13T07:47:37Z)
• Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruption [4.792851066169872]
  We find that vision-language models still suffer when they face datasets with large gaps from training ones.<n>We propose a novel method called information-balanced TTA (UnInfo) to make models robust to sensor degradation.
  arXiv  Detail & Related papers  (2025-05-19T09:47:46Z)
• We Care Each Pixel: Calibrating on Medical Segmentation Model [12.592472854721333]
  pixel-wise Expected Error (pECE) is a novel metric that measures miscalibration at the pixel level.<n>We also introduce a morphological adaptation strategy that applies morphological operations to ground-truth masks before computing calibration losses.<n>Our method not only enhances segmentation performance but also improves calibration quality, yielding more trustworthy confidence estimates.
  arXiv  Detail & Related papers  (2025-03-07T03:06:03Z)
• Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models [2.6061534894032907]
  We introduce textbfConfidence-Calibrated Covariate Shift Correction (CalShift) -- a unified approach that combines a Fisher information penalty to mitigate covariate shift and a Confidence Misalignment Penalty (CMP) to reduce overconfidence in misclassified examples.<n>CalShift significantly improves model calibration, achieving up to a 5.82% reduction in Expected Error (ECE)<n>Our results highlight CalShift as a promising strategy for building robust and reliable low-shot vision-language systems for real-world applications.
  arXiv  Detail & Related papers  (2025-02-11T10:10:15Z)
• Technical report on label-informed logit redistribution for better domain generalization in low-shot classification with foundation models [0.0]
  Confidence calibration is an emerging challenge in real-world decision systems based on foundations models.<n>We propose a penalty incorporated into loss objective that penalizes incorrect classifications whenever one is made during finetuning.<n>We refer to it as textitconfidence misalignment penalty (CMP).
  arXiv  Detail & Related papers  (2025-01-29T11:54:37Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation [15.802559423418804]
  We propose to use marginal L1 average calibration error (mL1-ACE) as a novel auxiliary loss function to improve pixel-wise calibration without compromising segmentation quality.<n>We show that this loss, despite using hard binning, is directly differentiable, bypassing the need for approximate but differentiable surrogate or soft binning approaches.
  arXiv  Detail & Related papers  (2024-03-11T14:31:03Z)
• Rethinking Few-shot 3D Point Cloud Semantic Segmentation [62.80639841429669]
  This paper revisits few-shot 3D point cloud semantic segmentation (FS-PCS) We focus on two significant issues in the state-of-the-art: foreground leakage and sparse point distribution. To address these issues, we introduce a standardized FS-PCS setting, upon which a new benchmark is built.
  arXiv  Detail & Related papers  (2024-03-01T15:14:47Z)
• On the Calibration of Human Pose Estimation [39.15814732856338]
  Calibrated ConfidenceNet (CCNet) is a light-weight post-hoc addition that improves AP by up to 1.4% on off-the-shelf pose estimation frameworks. applied to the downstream task of mesh recovery, CCNet facilitates an additional 1.0mm decrease in 3D keypoint error.
  arXiv  Detail & Related papers  (2023-11-28T09:31:09Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• Deep Metric Learning for Unsupervised Remote Sensing Change Detection [60.89777029184023]
  Remote Sensing Change Detection (RS-CD) aims to detect relevant changes from Multi-Temporal Remote Sensing Images (MT-RSIs) The performance of existing RS-CD methods is attributed to training on large annotated datasets. This paper proposes an unsupervised CD method based on deep metric learning that can deal with both of these issues.
  arXiv  Detail & Related papers  (2023-03-16T17:52:45Z)
• Pseudo Label-Guided Model Inversion Attack via Conditional Generative Adversarial Network [102.21368201494909]
  Model inversion (MI) attacks have raised increasing concerns about privacy. Recent MI attacks leverage a generative adversarial network (GAN) as an image prior to narrow the search space. We propose Pseudo Label-Guided MI (PLG-MI) attack via conditional GAN (cGAN)
  arXiv  Detail & Related papers  (2023-02-20T07:29:34Z)
• DOMINO: Domain-aware Model Calibration in Medical Image Segmentation [51.346121016559024]
  Modern deep neural networks are poorly calibrated, compromising trustworthiness and reliability. We propose DOMINO, a domain-aware model calibration method that leverages the semantic confusability and hierarchical similarity between class labels. Our results show that DOMINO-calibrated deep neural networks outperform non-calibrated models and state-of-the-art morphometric methods in head image segmentation.
  arXiv  Detail & Related papers  (2022-09-13T15:31:52Z)
• A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration [12.449806152650657]
  We propose a novel auxiliary loss function: Multi-class Difference in Confidence and Accuracy ( MDCA ) We show that training with MDCA leads to better-calibrated models in terms of Expected Error ( ECE), and Static Error ( SCE ) on image classification, and segmentation tasks.
  arXiv  Detail & Related papers  (2022-03-25T18:02:13Z)
• Test-time Batch Statistics Calibration for Covariate Shift [66.7044675981449]
  We propose to adapt the deep models to the novel environment during inference. We present a general formulation $alpha$-BN to calibrate the batch statistics. We also present a novel loss function to form a unified test time adaptation framework Core.
  arXiv  Detail & Related papers  (2021-10-06T08:45:03Z)
• Uncertainty-Aware Camera Pose Estimation from Points and Lines [101.03675842534415]
  Perspective-n-Point-and-Line (Pn$PL) aims at fast, accurate and robust camera localizations with respect to a 3D model from 2D-3D feature coordinates.
  arXiv  Detail & Related papers  (2021-07-08T15:19:36Z)
• A Weakly Supervised Consistency-based Learning Method for COVID-19 Segmentation in CT Images [11.778195406694206]
  Coronavirus Disease 2019 (COVID-19) has spread aggressively across the world causing an existential health crisis. A system that automatically detects COVID-19 in tomography (CT) images can assist in quantifying the severity of the illness. We address these labelling challenges by only requiring point annotations, a single pixel for each infected region on a CT image.
  arXiv  Detail & Related papers  (2020-07-04T20:41:17Z)
• Collaborative Boundary-aware Context Encoding Networks for Error Map Prediction [65.44752447868626]
  We propose collaborative boundaryaware context encoding networks called AEP-Net for error prediction task. Specifically, we propose a collaborative feature transformation branch for better feature fusion between images and masks, and precise localization of error regions. The AEP-Net achieves an average DSC of 0.8358, 0.8164 for error prediction task, and shows a high Pearson correlation coefficient of 0.9873.
  arXiv  Detail & Related papers  (2020-06-25T12:42:01Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.