MedFM-Robust: Benchmarking Robustness of Medical Foundation Models

• URL: http://arxiv.org/abs/2605.19027v3
• Date: Fri, 22 May 2026 22:22:01 GMT
• Title: MedFM-Robust: Benchmarking Robustness of Medical Foundation Models
• Abstract summary: We present a robustness benchmark comprising 40 perturbation types across eight imaging modalities.<n>Fine-tuning strategy dominates robustness, with LoRA exhibiting nearly double the degradation of full fine-tuning.<n>Medical-specific perturbations disproportionately damage segmentation, with 9 of 15 top corruptions being domain-specific.
• Score: 17.68454119152175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Medical foundation models have achieved remarkable clinical performance, yet their robustness under real-world perturbations remains underexplored. We present a robustness benchmark comprising 40 perturbation types (12 base, 28 medical-specific) across eight imaging modalities, evaluating five VLMs (LLaVA-Med, MedGemma, MedGemma-1.5, Gemini-2.5-flash and GPT-4o-mini) on VQA, visual grounding, and captioning, alongside two segmentation models (MedSAM, SAM-Med2D) with five fine-tuning strategies. Our findings reveal: (1) Fine-tuning strategy dominates robustness, with LoRA exhibiting nearly double the degradation of full fine-tuning, while SAM-Med2D's Adapter offers favorable efficiency-robustness trade-off. (2) Medical-specific perturbations disproportionately damage segmentation, with 9 of 15 top corruptions being domain-specific. (3) LoRA-tuned visual grounding drops over 40 points, whereas zero-shot captioning remains stable (<7% drop). Zero-shot VQA shows model-dependent robustness--medical models drop under 20% while Gemini-2.5-flash drops 54%. General-purpose VLMs achieve higher VQA accuracy but fail on grounding; among medical VLMs, MedGemma demonstrates the best overall stability. These results provide deployment guidelines and underscore the necessity of domain-specific robustness evaluation for medical AI. Our code is available at: https://abnerai.github.io/MedFM-Robust.

Related papers

• Auditing Frontier Vision-Language Models for Trustworthy Medical VQA: Grounding Failures, Format Collapse, and Domain Adaptation [4.177028541321909]
  We audit five vision-language models (VLMs) on Medical VQA along two trust-relevant axes.<n>Models localize anatomical and pathological targets poorly.<n>A self-grounding pipeline, where the same model localizes then answers, degrades VQA accuracy.
  arXiv  Detail & Related papers  (2026-04-30T11:11:47Z)
• Extending ZACH-ViT to Robust Medical Imaging: Corruption and Adversarial Stress Testing in Low-Data Regimes [0.23166812624868952]
  ZACH-ViT (Zero-token Adaptive Compact Hierarchical Vision Transformer) formalized a compact permutation-invariant Vision Transformer for medical imaging.<n>We present the first robustness-focused extension of ZACH-ViT by evaluating its behavior under common image corruptions and adversarial perturbations.
  arXiv  Detail & Related papers  (2026-04-07T17:11:07Z)
• SafeMed-R1: Adversarial Reinforcement Learning for Generalizable and Robust Medical Reasoning in Vision-Language Models [0.0]
  We introduce SafeMed-R1, a hybrid defense framework that ensures robust performance while preserving high-quality, interpretable medical reasoning.<n>We demonstrate that models trained with explicit chain-of-thought reasoning exhibit superior adversarial robustness compared to instruction-only variants.
  arXiv  Detail & Related papers  (2025-12-22T12:07:33Z)
• MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding [47.843626983298726]
  We introduce textbfMedVidBench, a large-scale benchmark of 531,850 video-instruction pairs across 8 medical sources spanning video, segment, and frame-level tasks.<n>While supervised fine-tuning on MedVidBench yields noticeable gains, standard Reinforcement Learning fails due to imbalanced reward scales across datasets.<n>We introduce textbfMedGRPO, a novel RL framework for balanced multi-dataset training with two key innovations.
  arXiv  Detail & Related papers  (2025-12-06T22:27:59Z)
• DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models [59.45605332033458]
  Safety mechanisms can backfire, causing over-refusal, where models decline benign requests out of excessive caution.<n>No existing benchmark has systematically addressed over-refusal in the visual modality.<n>This setting introduces unique challenges, such as dual-use cases where an instruction is harmless, but the accompanying image contains harmful content.
  arXiv  Detail & Related papers  (2025-10-12T23:21:34Z)
• EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models [82.43729208063468]
  Recent benchmarks for medical Large Vision-Language Models (LVLMs) emphasize leaderboard accuracy, overlooking reliability and safety.<n>We study sycophancy -- models' tendency to uncritically echo user-provided information.<n>We introduce EchoBench, a benchmark to systematically evaluate sycophancy in medical LVLMs.
  arXiv  Detail & Related papers  (2025-09-24T14:09:55Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation [21.766481181140527]
  We propose MedSAMix, a training-free model merging method for medical image segmentation.<n>We show that MedSAMix consistently improves performance in both domain-specific accuracy and generalization.<n>For clinical applications, we develop two regimes to meet the demand of domain-specificity and generalizability.
  arXiv  Detail & Related papers  (2025-08-14T19:35:57Z)
• MedGemma Technical Report [75.88152277443179]
  We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B.<n>MedGemma demonstrates advanced medical understanding and reasoning on images and text.<n>We additionally introduce MedSigLIP, a medically-tuned vision encoder derived from SigLIP.
  arXiv  Detail & Related papers  (2025-07-07T17:01:44Z)
• On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable? [0.9626666671366837]
  We introduce MediMeta-C, a corruption benchmark that applies several perturbations across multiple medical imaging datasets.<n>We propose RobustMedCLIP, a visual encoder adaptation of a pretrained MVLM that incorporates few-shot tuning to enhance resilience against corruptions.
  arXiv  Detail & Related papers  (2025-05-21T12:08:31Z)
• MedM-VL: What Makes a Good Medical LVLM? [14.598992440829079]
  Large vision-language models (LVLMs) offer new solutions for solving complex medical tasks.<n>We build on the popular LLaVA framework to explore model architectures and training strategies for both 2D and 3D medical LVLMs.<n>We release a modular, MedM-VL, and two pre-trained models: MedM-VL-2D for 2D medical image analysis and MedM-VL-CT-Chest for 3D CT-based applications.
  arXiv  Detail & Related papers  (2025-04-06T01:44:46Z)
• Biomedical Foundation Model: A Survey [84.26268124754792]
  Foundation models are large-scale pre-trained models that learn from extensive unlabeled datasets.<n>These models can be adapted to various applications such as question answering and visual understanding.<n>This survey explores the potential of foundation models across diverse domains within biomedical fields.
  arXiv  Detail & Related papers  (2025-03-03T22:42:00Z)
• Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST [7.017817009055001]
  We study the capabilities of foundation models in medical image classification tasks by conducting a benchmark study on the MedMNIST dataset.<n>We adopt various foundation models ranging from convolutional to Transformer-based models and implement both end-to-end training and linear probing for all classification tasks.
  arXiv  Detail & Related papers  (2025-01-24T18:01:07Z)
• Repurposing Foundation Model for Generalizable Medical Time Series Classification [16.21546283978257]
  FORMED is a framework for repurposing a backbone foundation model to enable highly generalizable MedTS classification on unseen datasets.<n>We evaluate FORMED on 5 diverse MedTS datasets, benchmarking against 11 Task-Specific Models (TSM) and 4 Task-Specific Adaptation (TSA) methods.<n>Our results demonstrate FORMED's dominant performance, achieving up to 35% absolute improvement in F1-score (on ADFTD dataset) over specialized baselines.
  arXiv  Detail & Related papers  (2024-10-03T23:50:04Z)
• CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models [92.04812189642418]
  We introduce CARES and aim to evaluate the Trustworthiness of Med-LVLMs across the medical domain. We assess the trustworthiness of Med-LVLMs across five dimensions, including trustfulness, fairness, safety, privacy, and robustness.
  arXiv  Detail & Related papers  (2024-06-10T04:07:09Z)
• Capabilities of Gemini Models in Medicine [100.60391771032887]
  We introduce Med-Gemini, a family of highly capable multimodal models specialized in medicine. We evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art (SoTA) performance on 10 of them. Our results offer compelling evidence for Med-Gemini's potential, although further rigorous evaluation will be crucial before real-world deployment.
  arXiv  Detail & Related papers  (2024-04-29T04:11:28Z)
• Segment Any Medical Model Extended [39.80956010574076]
  We introduce SAMM Extended (SAMME), a platform that integrates new SAM variant models, adopts faster communication protocols, accommodates new interactive modes, and allows for fine-tuning of subcomponents of the models. These features can expand the potential of foundation models like SAM, and the results can be translated to applications such as image-guided therapy, mixed reality interaction, robotic navigation, and data augmentation.
  arXiv  Detail & Related papers  (2024-03-26T21:37:25Z)
• A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models [57.88111980149541]
  We introduce Asclepius, a novel Med-MLLM benchmark that assesses Med-MLLMs in terms of distinct medical specialties and different diagnostic capacities.<n>Grounded in 3 proposed core principles, Asclepius ensures a comprehensive evaluation by encompassing 15 medical specialties.<n>We also provide an in-depth analysis of 6 Med-MLLMs and compare them with 3 human specialists.
  arXiv  Detail & Related papers  (2024-02-17T08:04:23Z)
• Customizing General-Purpose Foundation Models for Medical Report Generation [64.31265734687182]
  The scarcity of labelled medical image-report pairs presents great challenges in the development of deep and large-scale neural networks. We propose customizing off-the-shelf general-purpose large-scale pre-trained models, i.e., foundation models (FMs) in computer vision and natural language processing.
  arXiv  Detail & Related papers  (2023-06-09T03:02:36Z)
• MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer [53.575573940055335]
  We propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2. We verify its effectiveness on 20 medical image segmentation tasks with different image modalities.
  arXiv  Detail & Related papers  (2023-01-19T03:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.