PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses

• URL: http://arxiv.org/abs/2603.13026v1
• Date: Fri, 13 Mar 2026 14:34:54 GMT
• Title: PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses
• Abstract summary: We propose PISmith, a framework that assesses existing prompt-injection defenses.<n>We find that applying standard GRPO to attack strong defenses leads to sub-optimal performance.<n>We introduce adaptive entropy regularization and dynamic advantage weighting to sustain exploration and amplify learning from scarce successes.
• Score: 19.85522880335499
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt injection poses serious security risks to real-world LLM applications, particularly autonomous agents. Although many defenses have been proposed, their robustness against adaptive attacks remains insufficiently evaluated, potentially creating a false sense of security. In this work, we propose PISmith, a reinforcement learning (RL)-based red-teaming framework that systematically assesses existing prompt-injection defenses by training an attack LLM to optimize injected prompts in a practical black-box setting, where the attacker can only query the defended LLM and observe its outputs. We find that directly applying standard GRPO to attack strong defenses leads to sub-optimal performance due to extreme reward sparsity -- most generated injected prompts are blocked by the defense, causing the policy's entropy to collapse before discovering effective attack strategies, while the rare successes cannot be learned effectively. In response, we introduce adaptive entropy regularization and dynamic advantage weighting to sustain exploration and amplify learning from scarce successes. Extensive evaluation on 13 benchmarks demonstrates that state-of-the-art prompt injection defenses remain vulnerable to adaptive attacks. We also compare PISmith with 7 baselines across static, search-based, and RL-based attack categories, showing that PISmith consistently achieves the highest attack success rates. Furthermore, PISmith achieves strong performance in agentic settings on InjecAgent and AgentDojo against both open-source and closed-source LLMs (e.g., GPT-4o-mini and GPT-5-nano). Our code is available at https://github.com/albert-y1n/PISmith.

Related papers

• AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM Agents [57.34566159148893]
  Indirect prompt injection (IPI) is a major security threat to LLM-powered agents.<n>We develop AutoDojo, an adaptive extension of AgentDojo that optimize IPI against a given defense.
  arXiv  Detail & Related papers  (2026-06-13T02:09:08Z)
• When Can You Poison Rewards? A Tight Characterization of Reward Poisoning in Linear MDPs [36.48167187368067]
  This paper provides the first precise necessity and sufficiency characterization of the attackability of a linear MDP under reward poisoning attacks.<n>By approximating deep RL environments as linear MDPs, we show that our theoretical framework effectively distinguishes the attackability and efficiently attacks the vulnerable ones.
  arXiv  Detail & Related papers  (2026-04-11T06:55:45Z)
• Invisible Threats from Model Context Protocol: Generating Stealthy Injection Payload via Tree-based Adaptive Search [19.570062155058135]
  Recent advances in the Model Context Protocol (MCP) have enabled large language models (LLMs) to invoke external tools with unprecedented ease.<n>This creates a new class of powerful and tool augmented agents.<n>Existing techniques for indirect prompt injection that target MCP suffer from high deployment costs, weak semantic coherence, or heavy white box requirements.<n>We propose Tree structured Injection for Payloads (TIP), a novel black-box attack which generates natural payloads to reliably seize control of MCP enabled agents even under defense.
  arXiv  Detail & Related papers  (2026-03-25T11:24:47Z)
• RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection [82.41836544860833]
  We introduce RL-Hammer, a simple recipe for training attacker models that automatically learn to perform strong prompt injections.<n>We propose a set of practical techniques that enable highly effective, universal attacks.<n> RL-Hammer reaches a 98% ASR against GPT-4o and a $72%$ ASR against GPT-5 with the Instruction Hierarchy defense.
  arXiv  Detail & Related papers  (2025-10-06T15:06:04Z)
• Is Your Prompt Safe? Investigating Prompt Injection Attacks Against Open-Source LLMs [28.75283403986172]
  Large Language Models (LLMs) are vulnerable to prompt-based attacks, generating harmful content or sensitive information.<n>This paper studies effective prompt injection attacks against the $mathbf14$ most popular open-source LLMs on five attack benchmarks.
  arXiv  Detail & Related papers  (2025-05-20T13:50:43Z)
• CheatAgent: Attacking LLM-Empowered Recommender Systems via LLM Agent [32.958798200220286]
  Large Language Model (LLM)-empowered recommender systems (RecSys) have brought significant advances in personalized user experience.<n>We propose a novel attack framework called CheatAgent by harnessing the human-like capabilities of LLMs.<n>Our method first identifies the insertion position for maximum impact with minimal input modification.
  arXiv  Detail & Related papers  (2025-04-13T05:31:37Z)
• ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Models [55.93380086403591]
  Generative large language models are vulnerable to backdoor attacks.<n>$textitELBA-Bench$ allows attackers to inject backdoor through parameter efficient fine-tuning.<n>$textitELBA-Bench$ provides over 1300 experiments.
  arXiv  Detail & Related papers  (2025-02-22T12:55:28Z)
• MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents [60.30753230776882]
  LLM agents are vulnerable to indirect prompt injection (IPI) attacks, where malicious tasks embedded in tool-retrieved information can redirect the agent to take unauthorized actions.<n>We present MELON, a novel IPI defense that detects attacks by re-executing the agent's trajectory with a masked user prompt modified through a masking function.
  arXiv  Detail & Related papers  (2025-02-07T18:57:49Z)
• Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment [35.344406718760574]
  A prompt injection attack aims to make an Large Language Model follow an injected prompt to perform an attacker-chosen task.<n>Existing attacks primarily focus on crafting these injections at inference time, treating the LLM itself as a static target.<n>In this work, we introduce a more foundational attack vector: poisoning the LLM's alignment process to amplify the success of future prompt injection attacks.
  arXiv  Detail & Related papers  (2024-10-18T18:52:16Z)
• Evaluating Defences against Unsafe Feedback in RLHF [26.872318173182414]
  This paper looks at learning from unsafe feedback with reinforcement learning.<n>We find that safety-aligned LLMs easily explore unsafe action spaces via generating harmful text.<n>In order to protect against this vulnerability, we adapt a number of both "implict" and "explicit" harmful fine-tuning defences.
  arXiv  Detail & Related papers  (2024-09-19T17:10:34Z)
• PAL: Proxy-Guided Black-Box Attack on Large Language Models [55.57987172146731]
  Large Language Models (LLMs) have surged in popularity in recent months, but they have demonstrated capabilities to generate harmful content when manipulated. We introduce the Proxy-Guided Attack on LLMs (PAL), the first optimization-based attack on LLMs in a black-box query-only setting. Our attack achieves 84% attack success rate (ASR) on GPT-3.5-Turbo and 48% on Llama-2-7B, compared to 4% for the current state of the art.
  arXiv  Detail & Related papers  (2024-02-15T02:54:49Z)
• Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models [79.0183835295533]
  We introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities.<n>Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content.<n>We propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings.
  arXiv  Detail & Related papers  (2023-12-21T01:08:39Z)
• Formalizing and Benchmarking Prompt Injection Attacks and Defenses [59.57908526441172]
  We propose a framework to formalize prompt injection attacks. Based on our framework, we design a new attack by combining existing ones. Our work provides a common benchmark for quantitatively evaluating future prompt injection attacks and defenses.
  arXiv  Detail & Related papers  (2023-10-19T15:12:09Z)
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations [88.94162416324505]
  A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises. Since the observations deviate from the true states, they can mislead the agent into making suboptimal actions. We show that naively applying existing techniques on improving robustness for classification tasks, like adversarial training, is ineffective for many RL tasks.
  arXiv  Detail & Related papers  (2020-03-19T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.