Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation

• URL: http://arxiv.org/abs/2505.02971v1
• Date: Mon, 05 May 2025 18:54:41 GMT
• Title: Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation
• Authors: Anjila Budathoki, Manish Dhakal,
• Abstract summary: Adrial attacks have been fairly explored for computer vision and vision-language models.<n>We have investigated the robustness of vision language segmentation models (VLSMs) against adversarial attacks for 2D medical images.<n>The results exhibit significant drops in the DSC and IoU scores after the introduction of these adversaries.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial attacks have been fairly explored for computer vision and vision-language models. However, the avenue of adversarial attack for the vision language segmentation models (VLSMs) is still under-explored, especially for medical image analysis. Thus, we have investigated the robustness of VLSMs against adversarial attacks for 2D medical images with different modalities with radiology, photography, and endoscopy. The main idea of this project was to assess the robustness of the fine-tuned VLSMs specially in the medical domain setting to address the high risk scenario. First, we have fine-tuned pre-trained VLSMs for medical image segmentation with adapters. Then, we have employed adversarial attacks -- projected gradient descent (PGD) and fast gradient sign method (FGSM) -- on that fine-tuned model to determine its robustness against adversaries. We have reported models' performance decline to analyze the adversaries' impact. The results exhibit significant drops in the DSC and IoU scores after the introduction of these adversaries. Furthermore, we also explored universal perturbation but were not able to find for the medical images. \footnote{https://github.com/anjilab/secure-private-ai}

Related papers

• Hierarchical Self-Supervised Adversarial Training for Robust Vision Models in Histopathology [64.46054930696052]
  Adversarial attacks pose significant challenges for vision models in critical fields like healthcare.<n>Existing self-supervised adversarial training methods overlook the hierarchical structure of histopathology images.<n>We propose Hierarchical Self-Supervised Adversarial Training (HSAT), which exploits these properties to craft adversarial examples.
  arXiv  Detail & Related papers  (2025-03-13T17:59:47Z)
• Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector [97.92369017531038]
  We build a new laRge-scale Adervsarial images dataset with Diverse hArmful Responses (RADAR) We then develop a novel iN-time Embedding-based AdveRSarial Image DEtection (NEARSIDE) method, which exploits a single vector that distilled from the hidden states of Visual Language Models (VLMs) to achieve the detection of adversarial images against benign ones in the input.
  arXiv  Detail & Related papers  (2024-10-30T10:33:10Z)
• Adversary-Robust Graph-Based Learning of WSIs [2.9998889086656586]
  Whole slide images (WSIs) are high-resolution, digitized versions of tissue samples mounted on glass slides, scanned using sophisticated imaging equipment. The digital analysis of WSIs presents unique challenges due to their gigapixel size and multi-resolution storage format. We develop a novel and innovative graph-based model which utilizes GNN to extract features from the graph representation of WSIs.
  arXiv  Detail & Related papers  (2024-03-21T15:37:37Z)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images [68.42215385041114]
  This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-19T09:28:19Z)
• Adversarial-Robust Transfer Learning for Medical Imaging via Domain Assimilation [17.46080957271494]
  The scarcity of publicly available medical images has led contemporary algorithms to depend on pretrained models grounded on a large set of natural images. A significant em domain discrepancy exists between natural and medical images, which causes AI models to exhibit heightened em vulnerability to adversarial attacks. This paper proposes a em domain assimilation approach that introduces texture and color adaptation into transfer learning, followed by a texture preservation component to suppress undesired distortion.
  arXiv  Detail & Related papers  (2024-02-25T06:39:15Z)
• MITS-GAN: Safeguarding Medical Imaging from Tampering with Generative Adversarial Networks [48.686454485328895]
  This study introduces MITS-GAN, a novel approach to prevent tampering in medical images. The approach disrupts the output of the attacker's CT-GAN architecture by introducing finely tuned perturbations that are imperceptible to the human eye. Experimental results on a CT scan demonstrate MITS-GAN's superior performance.
  arXiv  Detail & Related papers  (2024-01-17T22:30:41Z)
• Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging [1.279856000554626]
  This study explores the vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a Vision Language Foundation model, under targeted attacks. We employ Projected Gradient Descent (PGD) adversarial perturbation attacks to induce misclassifications intentionally. The study emphasizes the pressing need for robust defenses to ensure the reliability of AI models.
  arXiv  Detail & Related papers  (2024-01-04T22:49:15Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• Survey on Adversarial Attack and Defense for Medical Image Analysis: Methods and Challenges [64.63744409431001]
  We present a comprehensive survey on advances in adversarial attacks and defenses for medical image analysis. For a fair comparison, we establish a new benchmark for adversarially robust medical diagnosis models.
  arXiv  Detail & Related papers  (2023-03-24T16:38:58Z)
• FIBA: Frequency-Injection based Backdoor Attack in Medical Image Analysis [82.2511780233828]
  We propose a novel Frequency-Injection based Backdoor Attack method (FIBA) that is capable of delivering attacks in various medical image analysis tasks. Specifically, FIBA leverages a trigger function in the frequency domain that can inject the low-frequency information of a trigger image into the poisoned image by linearly combining the spectral amplitude of both images.
  arXiv  Detail & Related papers  (2021-12-02T11:52:17Z)
• Towards Robust General Medical Image Segmentation [2.127049691404299]
  We propose a new framework to assess the robustness of general medical image segmentation systems. We present a novel lattice architecture for RObust Generic medical image segmentation (ROG) Our results show that ROG is capable of generalizing across different tasks of the MSD and largely surpasses the state-of-the-art under sophisticated adversarial attacks.
  arXiv  Detail & Related papers  (2021-07-09T07:17:05Z)
• Jekyll: Attacking Medical Image Diagnostics using Deep Generative Models [8.853343040790795]
  Jekyll is a neural style transfer framework that takes as input a biomedical image of a patient and translates it to a new image that indicates an attacker-chosen disease condition. We show that these attacks manage to mislead both medical professionals and algorithmic detection schemes. We also investigate defensive measures based on machine learning to detect images generated by Jekyll.
  arXiv  Detail & Related papers  (2021-04-05T18:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.