Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems

• URL: http://arxiv.org/abs/2410.07283v1
• Date: Wed, 9 Oct 2024 11:01:29 GMT
• Title: Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
• Authors: Donghyun Lee, Mo Tiwari,
• Abstract summary: We introduce Prompt Infection, a novel attack where malicious prompts self-replicate across interconnected agents. This attack poses severe threats, including data theft, scams, misinformation, and system-wide disruption. To address this, we propose LLM Tagging, a defense mechanism that, when combined with existing safeguards, significantly mitigates infection spread.
• Score: 6.480532634073257
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: As Large Language Models (LLMs) grow increasingly powerful, multi-agent systems are becoming more prevalent in modern AI applications. Most safety research, however, has focused on vulnerabilities in single-agent LLMs. These include prompt injection attacks, where malicious prompts embedded in external content trick the LLM into executing unintended or harmful actions, compromising the victim's application. In this paper, we reveal a more dangerous vector: LLM-to-LLM prompt injection within multi-agent systems. We introduce Prompt Infection, a novel attack where malicious prompts self-replicate across interconnected agents, behaving much like a computer virus. This attack poses severe threats, including data theft, scams, misinformation, and system-wide disruption, all while propagating silently through the system. Our extensive experiments demonstrate that multi-agent systems are highly susceptible, even when agents do not publicly share all communications. To address this, we propose LLM Tagging, a defense mechanism that, when combined with existing safeguards, significantly mitigates infection spread. This work underscores the urgent need for advanced security measures as multi-agent LLM systems become more widely adopted.

Related papers

• Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach [9.483655213280738]
  This paper presents a novel approach to evaluating the security of large language models (LLMs) We define prompt leakage as a critical threat to secure LLM deployment. We implement a multi-agent system where cooperative agents are tasked with probing and exploiting the target LLM to elicit its prompt.
  arXiv  Detail & Related papers  (2025-02-18T08:17:32Z)
• Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks [88.84977282952602]
  A high volume of recent ML security literature focuses on attacks against aligned large language models (LLMs) In this paper, we analyze security and privacy vulnerabilities that are unique to LLM agents. We conduct a series of illustrative attacks on popular open-source and commercial agents, demonstrating the immediate practical implications of their vulnerabilities.
  arXiv  Detail & Related papers  (2025-02-12T17:19:36Z)
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents [84.96249955105777]
  LLM agents may pose a greater risk if misused, but their robustness remains underexplored. We propose a new benchmark called AgentHarm to facilitate research on LLM agent misuse. We find leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking.
  arXiv  Detail & Related papers  (2024-10-11T17:39:22Z)
• Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context [49.13497493053742]
  This research explores converting a nonsensical suffix attack into a sensible prompt via a situation-driven contextual re-writing. We combine an independent, meaningful adversarial insertion and situations derived from movies to check if this can trick an LLM. Our approach demonstrates that a successful situation-driven attack can be executed on both open-source and proprietary LLMs.
  arXiv  Detail & Related papers  (2024-07-19T19:47:26Z)
• Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities [28.244283407749265]
  We investigate the security implications of large language models (LLMs) in multi-agent systems. We propose a novel two-stage attack method involving Persuasiveness Injection and Manipulated Knowledge Injection. We demonstrate that our attack method can successfully induce LLM-based agents to spread both counterfactual and toxic knowledge.
  arXiv  Detail & Related papers  (2024-07-10T16:08:46Z)
• Prompt Leakage effect and defense strategies for multi-turn LLM interactions [95.33778028192593]
  Leakage of system prompts may compromise intellectual property and act as adversarial reconnaissance for an attacker. We design a unique threat model which leverages the LLM sycophancy effect and elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. We measure the mitigation effect of 7 black-box defense strategies, along with finetuning an open-source model to defend against leakage attempts.
  arXiv  Detail & Related papers  (2024-04-24T23:39:58Z)
• The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operative [55.08395463562242]
  Multimodal Large Language Models (MLLMs) are constantly defining the new boundary of Artificial General Intelligence (AGI) Our paper explores a novel vulnerability in MLLM societies - the indirect propagation of malicious content.
  arXiv  Detail & Related papers  (2024-02-20T23:08:21Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.