PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections

• URL: http://arxiv.org/abs/2606.12737v1
• Date: Wed, 10 Jun 2026 22:57:02 GMT
• Title: PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections
• Abstract summary: Large Language Models (LLMs) are rapidly evolving into agentic systems that interact with external tools and environments.<n>We propose PI-Hunter, an automated agentic auditing framework for proactive vulnerability exposure in LLM agents.
• Score: 51.8246149916068
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are rapidly evolving into agentic systems that interact with external tools and environments, introducing new security risks such as indirect prompt injection attacks through untrusted external sources. Existing defenses mainly focus on blocking malicious content at inference time, and current red-teaming methods primarily optimize attack success. As a result, developers have limited visibility into how latent prompt injections emerge and propagate through agents. We propose PI-Hunter, an automated agentic auditing framework for proactive vulnerability exposure in LLM agents. PI-Hunter constructs realistic source-aware test cases and iteratively evolves them through feedback-driven exploration to induce agents to retrieve and reveal latent malicious instructions embedded within external environments. Extensive experiments across multiple benchmarks, agent architectures, attacks, and defenses demonstrate that PI-Hunter substantially improves vulnerability exposure and attack-surface coverage over strong automated red-teaming baselines, while remaining effective under existing prompt injection defenses.

Related papers

• Linguistic Firewall: Geometry as Defense in Multi-Agent Systems Routing [1.4033701678475552]
  We introduce ANTAP, an evaluation-driven routing architecture that discards indirect proxies in favor of active capability testing.<n>ANTAP achieves near-zero ASR against description-based injection attacks, compared to 67.3% and above for the description-based router baseline.
  arXiv  Detail & Related papers  (2026-06-29T16:51:06Z)
• DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents [121.77550256034]
  DecodingTrust-Agent Platform (DTap) is a controllable and interactive red-teaming platform for AI agents.<n>DTap-Red is the first autonomous red-teaming agent that explores diverse injection vectors and autonomously discovers effective attack strategies.<n>Through DTap, we conduct large-scale evaluations of popular AI agents built on various backbone models.
  arXiv  Detail & Related papers  (2026-05-06T11:59:48Z)
• AgentVisor: Defending LLM Agents Against Prompt Injection via Semantic Virtualization [37.5130864321051]
  We propose AgentVisor, a novel defense framework that enforces semantic privilege separation.<n>AgentVisor treats the target agent as an untrusted guest and intercepts tool calls via a trusted semantic visor.<n>We show that AgentVisor reduces the attack success rate to 0.65%, achieving this strong defense while incurring only a 1.45% average decrease in utility.
  arXiv  Detail & Related papers  (2026-04-27T07:12:52Z)
• Your Agent is More Brittle Than You Think: Uncovering Indirect Injection Vulnerabilities in Agentic LLMs [32.38053469964495]
  Extended action spaces, including hidden inter-system interactions, pose severe security challenges.<n>IPI, which conceals malicious instructions within third-party content, can trigger unauthorized actions such as data exfiltration.<n>We evaluate six defense strategies against four sophisticated IPI attack vectors across nine LLM backbones.
  arXiv  Detail & Related papers  (2026-04-04T21:27:04Z)
• T-MAP: Red-Teaming LLM Agents with Trajectory-aware Evolutionary Search [49.99459363244884]
  We propose a trajectory-aware evolutionary search method, T-MAP, which leverages execution trajectories to guide the discovery of adversarial prompts.<n>Our approach enables the automatic generation of attacks that not only bypass safety guardrails but also reliably realize harmful objectives through actual tool interactions.
  arXiv  Detail & Related papers  (2026-03-21T12:33:34Z)
• Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs [65.6660735371212]
  We present textbftextscJustAsk, a framework that autonomously discovers effective extraction strategies through interaction alone.<n>It formulates extraction as an online exploration problem, using Upper Confidence Bound--based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration.<n>Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.
  arXiv  Detail & Related papers  (2026-01-29T03:53:25Z)
• It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents [52.81924177620322]
  Web-based agents powered by large language models are increasingly used for tasks such as email management or professional networking.<n>Their reliance on dynamic web content makes them vulnerable to prompt injection attacks: adversarial instructions hidden in interface elements that persuade the agent to divert from its original task.<n>We introduce the Task-Redirecting Agent Persuasion Benchmark (TRAP), an evaluation for studying how persuasion techniques misguide autonomous web agents on realistic tasks.
  arXiv  Detail & Related papers  (2025-12-29T01:09:10Z)
• Exploiting Web Search Tools of AI Agents for Data Exfiltration [0.46664938579243564]
  Large language models (LLMs) are now routinely used to execute complex tasks, from natural language processing to dynamic like web searches.<n>The usage of tool-calling and Retrieval Augmented Generation (RAG) allows LLMs to process and retrieve sensitive corporate data, amplifying both their functionality and vulnerability to abuse.<n>We analyze how susceptible current LLMs are to indirect prompt injection attacks, which parameters, including model size and manufacturer, shape their vulnerability, and which attack methods remain most effective.
  arXiv  Detail & Related papers  (2025-10-10T07:39:01Z)
• IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents [33.775221377823925]
  Large language model (LLM) agents are widely deployed in real-world applications, where they leverage tools to retrieve and manipulate external data for complex tasks.<n>When interacting with untrusted data sources, tool responses may contain injected instructions that covertly influence agent behaviors and lead to malicious outcomes.<n>We propose a novel defensive task execution paradigm, called IPIGuard, to prevent malicious tool invocations at the source.
  arXiv  Detail & Related papers  (2025-08-21T07:08:16Z)
• AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents [54.29555239363013]
  We propose a generic black-box fuzzing framework, AgentVigil, to automatically discover and exploit indirect prompt injection vulnerabilities.<n>We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o.<n>We apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.
  arXiv  Detail & Related papers  (2025-05-09T07:40:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.