When Background Matters: Breaking Medical Vision Language Models by Transferable Attack

• URL: http://arxiv.org/abs/2604.17318v1
• Date: Sun, 19 Apr 2026 08:17:43 GMT
• Title: When Background Matters: Breaking Medical Vision Language Models by Transferable Attack
• Abstract summary: Vision-Language Models (VLMs) are increasingly used in clinical diagnostics, yet their robustness to adversarial attacks remains largely unexplored.<n>We propose MedFocusLeak, a highly transferable black-box multimodal attack that induces incorrect yet clinically plausible diagnoses.
• Score: 32.17770161701116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) are increasingly used in clinical diagnostics, yet their robustness to adversarial attacks remains largely unexplored, posing serious risks. Existing medical attacks focus on secondary objectives such as model stealing or adversarial fine-tuning, while transferable attacks from natural images introduce visible distortions that clinicians can easily detect. To address this, we propose MedFocusLeak, a highly transferable black-box multimodal attack that induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible. The method injects coordinated perturbations into non-diagnostic background regions and employs an attention distraction mechanism to shift the model's focus away from pathological areas. Extensive evaluations across six medical imaging modalities show that MedFocusLeak achieves state-of-the-art performance, generating misleading yet realistic diagnostic outputs across diverse VLMs. We further introduce a unified evaluation framework with novel metrics that jointly capture attack success and image fidelity, revealing a critical weakness in the reasoning capabilities of modern clinical VLMs.

Related papers

• Dialectic-Med: Mitigating Diagnostic Hallucinations via Counterfactual Adversarial Multi-Agent Debate [2.645243404278707]
  We propose Dialectic-Med, a multi-agent framework that enforces diagnostic rigor through adversarial dialectics.<n>By explicitly modeling the cognitive process of falsification, our framework guarantees that diagnostic reasoning is tightly grounded in verified visual regions.
  arXiv  Detail & Related papers  (2026-04-13T10:09:36Z)
• MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence [17.512220295924816]
  Current medical Vision-Language Models (VLMs) still lack mechanisms that translate their broad knowledge into clinically actionable outputs.<n>We present MEDIC-AD, a clinically oriented VLM that strengthens these three capabilities through a stage-wise framework.<n>We show that MEDIC-AD steadily boosts performance across anomaly detection, symptom tracking, and anomaly segmentation.
  arXiv  Detail & Related papers  (2026-03-28T07:26:40Z)
• When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasound [17.261506344527195]
  Vision-language models (VLMs) have recently demonstrated strong multimodal reasoning capabilities and competitive performance in medical image analysis, including ultrasound.<n> adversarial robustness is critical because Med-VLMs operate via natural-language instructions, rendering prompt formulation a realistic and practically exploitable point of vulnerability.<n>We propose a scalable adversarial evaluation framework that leverages a large language model (LLM) to generate clinically plausible adversarial prompt variants.
  arXiv  Detail & Related papers  (2026-03-22T03:58:34Z)
• Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs [63.535652574541764]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis.<n>Their application in gastrointestinal endoscopy is currently hindered by two critical limitations.<n>We propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-21T07:47:37Z)
• On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable? [0.9626666671366837]
  We introduce MediMeta-C, a corruption benchmark that applies several perturbations across multiple medical imaging datasets.<n>We propose RobustMedCLIP, a visual encoder adaptation of a pretrained MVLM that incorporates few-shot tuning to enhance resilience against corruptions.
  arXiv  Detail & Related papers  (2025-05-21T12:08:31Z)
• CoRPA: Adversarial Image Generation for Chest X-rays Using Concept Vector Perturbations and Generative Models [2.380494879018844]
  Deep learning models for medical image classification tasks are becoming widely implemented in AI-assisted diagnostic tools.<n>Their vulnerability to adversarial attacks poses significant risks to patient safety.<n>We propose the Concept-based Report Perturbation Attack (CoRPA), a clinically-focused black-box adversarial attack framework.
  arXiv  Detail & Related papers  (2025-02-04T17:14:31Z)
• Natias: Neuron Attribution based Transferable Image Adversarial Steganography [62.906821876314275]
  adversarial steganography has garnered considerable attention due to its ability to effectively deceive deep-learning-based steganalysis. We propose a novel adversarial steganographic scheme named Natias. Our proposed method can be seamlessly integrated with existing adversarial steganography frameworks.
  arXiv  Detail & Related papers  (2024-09-08T04:09:51Z)
• Uncertainty-aware Medical Diagnostic Phrase Identification and Grounding [72.18719355481052]
  We introduce a novel task called Medical Report Grounding (MRG)<n>MRG aims to directly identify diagnostic phrases and their corresponding grounding boxes from medical reports in an end-to-end manner.<n>We propose uMedGround, a robust and reliable framework that leverages a multimodal large language model to predict diagnostic phrases.
  arXiv  Detail & Related papers  (2024-04-10T07:41:35Z)
• Gravity Network for end-to-end small lesion detection [50.38534263407915]
  This paper introduces a novel one-stage end-to-end detector specifically designed to detect small lesions in medical images. Precise localization of small lesions presents challenges due to their appearance and the diverse contextual backgrounds in which they are found. We refer to this new architecture as GravityNet, and the novel anchors as gravity points since they appear to be "attracted" by the lesions.
  arXiv  Detail & Related papers  (2023-09-22T14:02:22Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)
• An Interpretable Multiple-Instance Approach for the Detection of referable Diabetic Retinopathy from Fundus Images [72.94446225783697]
  We propose a machine learning system for the detection of referable Diabetic Retinopathy in fundus images. By extracting local information from image patches and combining it efficiently through an attention mechanism, our system is able to achieve high classification accuracy. We evaluate our approach on publicly available retinal image datasets, in which it exhibits near state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-03-02T13:14:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.