Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks

• URL: http://arxiv.org/abs/2403.03792v2
• Date: Thu, 2 May 2024 09:25:38 GMT
• Title: Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks
• Authors: Dario Pasquini, Martin Strohmeier, Carmela Troncoso,
• Abstract summary: We introduce a new family of prompt injection attacks, termed Neural Exec. We show that it is possible to conceptualize the creation of execution triggers as a differentiable search problem and use learning-based methods to autonomously generate them.
• Score: 20.058741696160798
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a new family of prompt injection attacks, termed Neural Exec. Unlike known attacks that rely on handcrafted strings (e.g., "Ignore previous instructions and..."), we show that it is possible to conceptualize the creation of execution triggers as a differentiable search problem and use learning-based methods to autonomously generate them. Our results demonstrate that a motivated adversary can forge triggers that are not only drastically more effective than current handcrafted ones but also exhibit inherent flexibility in shape, properties, and functionality. In this direction, we show that an attacker can design and generate Neural Execs capable of persisting through multi-stage preprocessing pipelines, such as in the case of Retrieval-Augmented Generation (RAG)-based applications. More critically, our findings show that attackers can produce triggers that deviate markedly in form and shape from any known attack, sidestepping existing blacklist-based detection and sanitation approaches.

Related papers

• Twin Trigger Generative Networks for Backdoor Attacks against Object Detection [14.578800906364414]
  Object detectors, which are widely used in real-world applications, are vulnerable to backdoor attacks. Most research on backdoor attacks has focused on image classification, with limited investigation into object detection. We propose novel twin trigger generative networks to generate invisible triggers for implanting backdoors into models during training, and visible triggers for steady activation during inference.
  arXiv  Detail & Related papers  (2024-11-23T03:46:45Z)
• Defense Against Prompt Injection Attack by Leveraging Attack Techniques [66.65466992544728]
  Large language models (LLMs) have achieved remarkable performance across various natural language processing (NLP) tasks. As LLMs continue to evolve, new vulnerabilities, especially prompt injection attacks arise. Recent attack methods leverage LLMs' instruction-following abilities and their inabilities to distinguish instructions injected in the data content.
  arXiv  Detail & Related papers  (2024-11-01T09:14:21Z)
• Rethinking Graph Backdoor Attacks: A Distribution-Preserving Perspective [33.35835060102069]
  Graph Neural Networks (GNNs) have shown remarkable performance in various tasks. Backdoor attack poisons the graph by attaching backdoor triggers and the target class label to a set of nodes in the training graph. In this paper, we study a novel problem of unnoticeable graph backdoor attacks with in-distribution (ID) triggers.
  arXiv  Detail & Related papers  (2024-05-17T13:09:39Z)
• VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models [65.23688155159398]
  Autoregressive Visual Language Models (VLMs) showcase impressive few-shot learning capabilities in a multimodal context. Recently, multimodal instruction tuning has been proposed to further enhance instruction-following abilities. Adversaries can implant a backdoor by injecting poisoned samples with triggers embedded in instructions or images. We propose a multimodal instruction backdoor attack, namely VL-Trojan.
  arXiv  Detail & Related papers  (2024-02-21T14:54:30Z)
• Pre-trained Trojan Attacks for Visual Recognition [106.13792185398863]
  Pre-trained vision models (PVMs) have become a dominant component due to their exceptional performance when fine-tuned for downstream tasks. We propose the Pre-trained Trojan attack, which embeds backdoors into a PVM, enabling attacks across various downstream vision tasks. We highlight the challenges posed by cross-task activation and shortcut connections in successful backdoor attacks.
  arXiv  Detail & Related papers  (2023-12-23T05:51:40Z)
• Versatile Backdoor Attack with Visible, Semantic, Sample-Specific, and Compatible Triggers [38.67988745745853]
  We propose a novel trigger called the textbfVisible, textbfSemantic, textbfSample-language, and textbfCompatible (VSSC) trigger. VSSC trigger achieves effective, stealthy and robust simultaneously, which can also be effectively deployed in the physical scenario using corresponding objects.
  arXiv  Detail & Related papers  (2023-06-01T15:42:06Z)
• Backdoor Attack with Sparse and Invisible Trigger [57.41876708712008]
  Deep neural networks (DNNs) are vulnerable to backdoor attacks. backdoor attack is an emerging yet threatening training-phase threat. We propose a sparse and invisible backdoor attack (SIBA)
  arXiv  Detail & Related papers  (2023-05-11T10:05:57Z)
• Invisible Backdoor Attack with Dynamic Triggers against Person Re-identification [71.80885227961015]
  Person Re-identification (ReID) has rapidly progressed with wide real-world applications, but also poses significant risks of adversarial attacks. We propose a novel backdoor attack on ReID under a new all-to-unknown scenario, called Dynamic Triggers Invisible Backdoor Attack (DT-IBA) We extensively validate the effectiveness and stealthiness of the proposed attack on benchmark datasets, and evaluate the effectiveness of several defense methods against our attack.
  arXiv  Detail & Related papers  (2022-11-20T10:08:28Z)
• Exploring the Universal Vulnerability of Prompt-based Learning Paradigm [21.113683206722207]
  We find that prompt-based learning bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. We explore this universal vulnerability by either injecting backdoor triggers or searching for adversarial triggers on pre-trained language models using only plain text.
  arXiv  Detail & Related papers  (2022-04-11T16:34:10Z)
• Poisoned classifiers are not only backdoored, they are fundamentally broken [84.67778403778442]
  Under a commonly-studied backdoor poisoning attack against classification models, an attacker adds a small trigger to a subset of the training data. It is often assumed that the poisoned classifier is vulnerable exclusively to the adversary who possesses the trigger. In this paper, we show empirically that this view of backdoored classifiers is incorrect.
  arXiv  Detail & Related papers  (2020-10-18T19:42:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.