AICAttack: Adversarial Image Captioning Attack with Attention-Based
Optimization
- URL: http://arxiv.org/abs/2402.11940v2
- Date: Tue, 20 Feb 2024 12:13:05 GMT
- Title: AICAttack: Adversarial Image Captioning Attack with Attention-Based
Optimization
- Authors: Jiyao Li, Mingze Ni, Yifei Dong, Tianqing Zhu and Wei Liu
- Abstract summary: We present a novel adversarial attack strategy, which we call AICAttack.
operating within a black-box attack scenario, our algorithm requires no access to the target model's architecture, parameters, or gradient information.
We demonstrate AICAttack's effectiveness through extensive experiments on benchmark datasets with multiple victim models.
- Score: 13.99541041673674
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recent advances in deep learning research have shown remarkable achievements
across many tasks in computer vision (CV) and natural language processing
(NLP). At the intersection of CV and NLP is the problem of image captioning,
where the related models' robustness against adversarial attacks has not been
well studied. In this paper, we present a novel adversarial attack strategy,
which we call AICAttack (Attention-based Image Captioning Attack), designed to
attack image captioning models through subtle perturbations on images.
Operating within a black-box attack scenario, our algorithm requires no access
to the target model's architecture, parameters, or gradient information. We
introduce an attention-based candidate selection mechanism that identifies the
optimal pixels to attack, followed by Differential Evolution (DE) for
perturbing pixels' RGB values. We demonstrate AICAttack's effectiveness through
extensive experiments on benchmark datasets with multiple victim models. The
experimental results demonstrate that our method surpasses current leading-edge
techniques by effectively distributing the alignment and semantics of words in
the output.
Related papers
- MirrorCheck: Efficient Adversarial Defense for Vision-Language Models [55.73581212134293]
We propose a novel, yet elegantly simple approach for detecting adversarial samples in Vision-Language Models.
Our method leverages Text-to-Image (T2I) models to generate images based on captions produced by target VLMs.
Empirical evaluations conducted on different datasets validate the efficacy of our approach.
arXiv Detail & Related papers (2024-06-13T15:55:04Z) - VQAttack: Transferable Adversarial Attacks on Visual Question Answering
via Pre-trained Models [58.21452697997078]
We propose a novel VQAttack model, which can generate both image and text perturbations with the designed modules.
Experimental results on two VQA datasets with five validated models demonstrate the effectiveness of the proposed VQAttack.
arXiv Detail & Related papers (2024-02-16T21:17:42Z) - SA-Attack: Improving Adversarial Transferability of Vision-Language
Pre-training Models via Self-Augmentation [56.622250514119294]
In contrast to white-box adversarial attacks, transfer attacks are more reflective of real-world scenarios.
We propose a self-augment-based transfer attack method, termed SA-Attack.
arXiv Detail & Related papers (2023-12-08T09:08:50Z) - OT-Attack: Enhancing Adversarial Transferability of Vision-Language
Models via Optimal Transport Optimization [65.57380193070574]
Vision-language pre-training models are vulnerable to multi-modal adversarial examples.
Recent works have indicated that leveraging data augmentation and image-text modal interactions can enhance the transferability of adversarial examples.
We propose an Optimal Transport-based Adversarial Attack, dubbed OT-Attack.
arXiv Detail & Related papers (2023-12-07T16:16:50Z) - A Black-box NLP Classifier Attacker [5.177150961252542]
We propose a word-level NLP sentiment classifier attack model, which includes a self-attention mechanism-based word selection method and a greedy search algorithm for word substitution.
Our model achieves a higher attack success rate and more efficient than previous methods due to the efficient word selection algorithms are employed and minimized the word substitute number.
arXiv Detail & Related papers (2021-12-22T04:25:23Z) - Geometrically Adaptive Dictionary Attack on Face Recognition [23.712389625037442]
We propose a strategy for query-efficient black-box attacks on face recognition.
Our core idea is to create an adversarial perturbation in the UV texture map and project it onto the face in the image.
We show overwhelming performance improvement in the experiments on the LFW and CPLFW datasets.
arXiv Detail & Related papers (2021-11-08T10:26:28Z) - Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial
Attack Framework [17.17479625646699]
We propose a unified framework to craft textual adversarial samples.
In this paper, we instantiate our framework with an attack algorithm named Textual Projected Gradient Descent (T-PGD)
arXiv Detail & Related papers (2021-10-28T17:31:51Z) - Deep Image Destruction: A Comprehensive Study on Vulnerability of Deep
Image-to-Image Models against Adversarial Attacks [104.8737334237993]
We present comprehensive investigations into the vulnerability of deep image-to-image models to adversarial attacks.
For five popular image-to-image tasks, 16 deep models are analyzed from various standpoints.
We show that unlike in image classification tasks, the performance degradation on image-to-image tasks can largely differ depending on various factors.
arXiv Detail & Related papers (2021-04-30T14:20:33Z) - PICA: A Pixel Correlation-based Attentional Black-box Adversarial Attack [37.15301296824337]
We propose a pixel correlation-based attentional black-box adversarial attack, termed as PICA.
PICA is more efficient to generate high-resolution adversarial examples compared with the existing black-box attacks.
arXiv Detail & Related papers (2021-01-19T09:53:52Z) - Reinforcement Learning-based Black-Box Evasion Attacks to Link
Prediction in Dynamic Graphs [87.5882042724041]
Link prediction in dynamic graphs (LPDG) is an important research problem that has diverse applications.
We study the vulnerability of LPDG methods and propose the first practical black-box evasion attack.
arXiv Detail & Related papers (2020-09-01T01:04:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.