MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models

• URL: http://arxiv.org/abs/2608.27004v2
• Date: Mon, 31 Aug 2026 12:39:07 GMT
• Title: MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models
• Abstract summary: MVCBench is a calibration-centric benchmark for medical image classification with vision-language models (VLMs) and medical vision-language models (Medical-VLMs)<n>It assesses the calibration across three axes: (i) robustness to modality, backbone, and domain shift (ii) effectiveness of calibration strategies and prompt-tuning methods (iii) stability under prompt-template and random-seed settings.<n>Across more than 1638 controlled experiments, we report accuracy and Expected Error (ECE) as primary metrics and further report results with complementary calibration measures.
• Score: 30.332542666039647
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable evaluation of vision-language models (VLMs) and medical vision-language models (Medical-VLMs) requires calibrated confidence, particularly under realistic clinical conditions. However, existing efforts mainly focused on improving accuracy, leaving calibration in the medical domain underexplored. To this end, we propose MVC-Bench, a calibration-centric benchmark for medical image classification with VLMs and Medical-VLMs. MVC-Bench assesses the calibration across three axes: (i) robustness to modality, backbone, and domain shift (ii) effectiveness of calibration strategies and prompt-tuning methods (iii) stability under prompt-template and random-seed variations. The benchmark covers eight different backbones, three medical modalities, including fundus imaging, histopathology, and chest X-ray under in-domain and domain shift settings. It compares post-hoc calibration, train-time calibration, and zero-shot inference methods, together with six prompt-tuning methods. Across more than 1638 controlled experiments, we report accuracy and Expected Calibration Error (ECE) as primary metrics, and further report results with complementary calibration measures, including Maximum Calibration Error (MCE) and Adaptive Calibration Error (ACE). We further investigate the underlying causes of miscalibration in VLMs and Medical-VLMs and propose a simple train-time calibration method, Multi-Class Margin (MCM) regularization, which achieves lowest ECE on 10 out of 12 settings in in-domain and remains competitive under domain shifts. Collectively, MVC-Bench provides a structured evaluation framework and actionable guidance for improving calibration in safety-critical medical workflows.

Related papers

• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigation [4.247706336083815]
  Overconfidence in vision-language models (VLMs) persists across model families and is not resolved by scaling or prompting.<n>Post-hoc calibration approaches, such as Platt scaling, reduce calibration error and consistently outperform the prompt-based strategy.<n> hallucination-aware calibration (HAC) incorporates vision-grounded hallucination detection signals as complementary inputs to refine confidence estimates.
  arXiv  Detail & Related papers  (2026-04-02T21:52:29Z)
• Scalable Utility-Aware Multiclass Calibration [53.28176049547449]
  Utility calibration is a general framework that measures the calibration error relative to a specific utility function.<n>We demonstrate how this framework can unify and re-interpret several existing calibration metrics.
  arXiv  Detail & Related papers  (2025-10-29T12:32:14Z)
• Calibration-Aware Prompt Learning for Medical Vision-Language Models [44.97741487992985]
  Miscalibrated predictions can lead to overconfident errors, undermining clinical trust and decision-making reliability.<n>We introduce CalibPrompt, the first framework to calibrate Med-VLMs during prompt tuning.<n>CalibPrompt consistently improves calibration without drastically affecting clean accuracy.
  arXiv  Detail & Related papers  (2025-09-18T17:59:58Z)
• Where are we with calibration under dataset shift in image classification? [13.432791129536255]
  We conduct a study on the state of calibration under real-world dataset shift for image classification.<n>We compare various post-hoc calibration methods, and their interactions with common in-training calibration strategies.<n>We find that: (i) applying calibration prior to ensembling is more effective for calibration under shifts.
  arXiv  Detail & Related papers  (2025-07-10T13:59:53Z)
• Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classification [8.43909252072479]
  Multimodal large language models (MLLMs) have enormous potential to perform few-shot in-context learning in the context of medical image analysis.<n>We present the first investigation into the calibration biases and demographic unfairness of MLLMs' predictions and confidence scores in few-shot in-context learning for medical image classification.<n>We introduce CALIN, an inference-time calibration method designed to mitigate the associated biases.
  arXiv  Detail & Related papers  (2025-06-29T15:37:17Z)
• Adaptive Set-Mass Calibration with Conformal Prediction [60.47079469141295]
  We develop a new calibration procedure that starts with conformal prediction to obtain a set of labels that gives the desired coverage.<n>We then instantiate two simple post-hoc calibrators: a mass normalization and a temperature scaling-based rule, tuned to the conformal constraint.
  arXiv  Detail & Related papers  (2025-05-21T12:18:15Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Calibration of Neural Networks [77.34726150561087]
  This paper presents a survey of confidence calibration problems in the context of neural networks. We analyze problem statement, calibration definitions, and different approaches to evaluation. Empirical experiments cover various datasets and models, comparing calibration methods according to different criteria.
  arXiv  Detail & Related papers  (2023-03-19T20:27:51Z)
• On Calibrating Semantic Segmentation Models: Analyses and An Algorithm [51.85289816613351]
  We study the problem of semantic segmentation calibration. Model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. We propose a simple, unifying, and effective approach, namely selective scaling.
  arXiv  Detail & Related papers  (2022-12-22T22:05:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.