Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging

• URL: http://arxiv.org/abs/2401.02565v3
• Date: Tue, 7 May 2024 18:14:42 GMT
• Title: Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging
• Authors: Poojitha Thota, Jai Prakash Veerla, Partha Sai Guttikonda, Mohammad S. Nasr, Shirin Nilizadeh, Jacob M. Luber,
• Abstract summary: This study explores the vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a Vision Language Foundation model, under targeted attacks. We employ Projected Gradient Descent (PGD) adversarial perturbation attacks to induce misclassifications intentionally. The study emphasizes the pressing need for robust defenses to ensure the reliability of AI models.
• Score: 1.279856000554626
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In the context of medical artificial intelligence, this study explores the vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a Vision Language Foundation model, under targeted attacks. Leveraging the Kather Colon dataset with 7,180 H&E images across nine tissue types, our investigation employs Projected Gradient Descent (PGD) adversarial perturbation attacks to induce misclassifications intentionally. The outcomes reveal a 100% success rate in manipulating PLIP's predictions, underscoring its susceptibility to adversarial perturbations. The qualitative analysis of adversarial examples delves into the interpretability challenges, shedding light on nuanced changes in predictions induced by adversarial manipulations. These findings contribute crucial insights into the interpretability, domain adaptation, and trustworthiness of Vision Language Models in medical imaging. The study emphasizes the pressing need for robust defenses to ensure the reliability of AI models. The source codes for this experiment can be found at https://github.com/jaiprakash1824/VLM_Adv_Attack.

Related papers

• Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation [0.0]
  Adrial attacks have been fairly explored for computer vision and vision-language models.<n>We have investigated the robustness of vision language segmentation models (VLSMs) against adversarial attacks for 2D medical images.<n>The results exhibit significant drops in the DSC and IoU scores after the introduction of these adversaries.
  arXiv  Detail & Related papers  (2025-05-05T18:54:41Z)
• SurvAttack: Black-Box Attack On Survival Models through Ontology-Informed EHR Perturbation [9.500873129276531]
  We introduce SurvAttack, a novel black-box adversarial attack framework for survival analysis models. We specifically develop an algorithm to manipulate medical codes with various adversarial actions throughout a patient's medical history. The proposed adversarial EHR perturbation algorithm is then used in an efficient SA-specific strategy to attack a survival model.
  arXiv  Detail & Related papers  (2024-12-24T23:35:42Z)
• MirrorCheck: Efficient Adversarial Defense for Vision-Language Models [55.73581212134293]
  We propose a novel, yet elegantly simple approach for detecting adversarial samples in Vision-Language Models. Our method leverages Text-to-Image (T2I) models to generate images based on captions produced by target VLMs. Empirical evaluations conducted on different datasets validate the efficacy of our approach.
  arXiv  Detail & Related papers  (2024-06-13T15:55:04Z)
• Adversarial-Robust Transfer Learning for Medical Imaging via Domain Assimilation [17.46080957271494]
  The scarcity of publicly available medical images has led contemporary algorithms to depend on pretrained models grounded on a large set of natural images. A significant em domain discrepancy exists between natural and medical images, which causes AI models to exhibit heightened em vulnerability to adversarial attacks. This paper proposes a em domain assimilation approach that introduces texture and color adaptation into transfer learning, followed by a texture preservation component to suppress undesired distortion.
  arXiv  Detail & Related papers  (2024-02-25T06:39:15Z)
• Application Of Vision-Language Models For Assessing Osteoarthritis Disease Severity [0.43431539537721414]
  Osteoarthritis (OA) poses a global health challenge, demanding precise diagnostic methods. Existing deep learning models for OA assessment are unimodal single task systems. This study investigates employing Vision Language Processing models to predict OA severity using Xray images and corresponding reports.
  arXiv  Detail & Related papers  (2024-01-12T02:43:58Z)
• SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation [56.622250514119294]
  In contrast to white-box adversarial attacks, transfer attacks are more reflective of real-world scenarios. We propose a self-augment-based transfer attack method, termed SA-Attack.
  arXiv  Detail & Related papers  (2023-12-08T09:08:50Z)
• A Survey on Transferability of Adversarial Examples across Deep Neural Networks [53.04734042366312]
  adversarial examples can manipulate machine learning models into making erroneous predictions. The transferability of adversarial examples enables black-box attacks which circumvent the need for detailed knowledge of the target model. This survey explores the landscape of the adversarial transferability of adversarial examples.
  arXiv  Detail & Related papers  (2023-10-26T17:45:26Z)
• Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models [49.95603725998561]
  We propose a new paradigm to build robust and interpretable medical image classifiers with natural language concepts. Specifically, we first query clinical concepts from GPT-4, then transform latent image features into explicit concepts with a vision-language model.
  arXiv  Detail & Related papers  (2023-10-04T21:57:09Z)
• Self-Ensembling Vision Transformer (SEViT) for Robust Medical Image Classification [4.843654097048771]
  Vision Transformers (ViT) are competing to replace Convolutional Neural Networks (CNN) for various computer vision tasks in medical imaging. Recent works have shown that ViTs are also susceptible to such attacks and suffer significant performance degradation under attack. We propose a novel self-ensembling method to enhance the robustness of ViT in the presence of adversarial attacks.
  arXiv  Detail & Related papers  (2022-08-04T19:02:24Z)
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models [103.64435911083432]
  We present a novel contrastive learning strategy called it Proactive Pseudo-Intervention (PPI) PPI leverages proactive interventions to guard against image features with no causal relevance. We also devise a novel causally informed salience mapping module to identify key image pixels to intervene, and show it greatly facilitates model interpretability.
  arXiv  Detail & Related papers  (2020-12-06T20:30:26Z)
• On the Transferability of Adversarial Attacksagainst Neural Text Classifier [121.6758865857686]
  We investigate the transferability of adversarial examples for text classification models. We propose a genetic algorithm to find an ensemble of models that can induce adversarial examples to fool almost all existing models. We derive word replacement rules that can be used for model diagnostics from these adversarial examples.
  arXiv  Detail & Related papers  (2020-11-17T10:45:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.