An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments

• URL: http://arxiv.org/abs/2605.18133v1
• Date: Mon, 18 May 2026 09:38:18 GMT
• Title: An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments
• Abstract summary: This paper studies a privacy-leakage attack chain based on indirect prompt injection.<n>We first analyze how an attacker can hijack an agent's intended task by crafting external content that appears benign to the victim.<n>We then evaluate a new prompt-language technique, called exemplification, which uses a bridge in the external content to reframe the user prompt and the benign beginning of the retrieved page.
• Score: 3.275342475988451
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-based chatbot agents increasingly process user requests by combining natural-language reasoning with external tools such as web browsing. These capabilities improve usability, but they also create attack surfaces when untrusted external content is processed as part of a user' s task. This paper studies a privacy-leakage attack chain based on indirect prompt injection in black-box chatbot environments, where the attacker has no access to model weights, system prompts, or agent implementation details including how a trajectory is actually managed during its processing for a query. We first analyze how an attacker can hijack an agent' s intended task by crafting external content that appears benign to the victim while inducing the agent to execute an attacker-defined objective. We then evaluate a new prompt-injection technique, called exemplification, which uses a bridge in the external content to reframe the user prompt and the benign beginning of the retrieved page as few-shot examples before appending the attacker' s objective. We compare its attack success rate with a prior fake-completion technique. Finally, we demonstrate a proof-of-concept data-exfiltration chain using fictitious personal information in a controlled setting. Our results suggest that prompt injection, jailbreak-style instruction steering, and web-tool invocation can be combined into a feasible privacy-leakage path in deployed chatbot agents.

Related papers

• Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection [40.37588773528427]
  We present Repeat-After-Me, a black-box adaptive visual prompt injection attack that can reveal personally identifiable information or make malicious tool calls.<n>In our evaluation, injections optimized on one surrogate retain 43-46% of the original ASR on two commercial victims.<n>In a default OpenClaw Discord deployment, an untrusted user can use a minimally injected image to overwrite TOOLS.md, enabling future sensitive behaviors like remote code execution and secret exfiltration.
  arXiv  Detail & Related papers  (2026-09-03T22:44:45Z)
• Invisible Ink Threats: Adversarial Goals Behind Legitimate Tasks in Computer-Use Agents [10.553545687886928]
  We present II-Bench, a collection of seemingly harmless adversarial tasks.<n>II-Bench comprises 444 examples targeting confidentiality and integrity attacks across three platforms.<n>We also construct HITLCUA, a comprehensive adversarial testing framework.
  arXiv  Detail & Related papers  (2026-08-03T10:16:53Z)
• Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents [64.80318459073526]
  We show that adversarial content can persist across interactions served by the same agent, making such threats harder to detect and mitigate.<n>Specifically, adversarial content may persist in the agent state, remain dormant across interactions, and later be activated by a benign user query.<n>We formalize this type of safety threat as Sleeper Attack. To evaluate it, we construct a benchmark with 1,896 instances covering six real-world harmful outcomes, three attack strategies, and three agent state targets.
  arXiv  Detail & Related papers  (2026-05-27T09:25:37Z)
• Trojan's Whisper: Stealthy Manipulation of OpenClaw through Injected Bootstrapped Guidance [23.059379933610163]
  Guidance injection is a stealthy attack vector that embeds adversarial operational narratives into bootstrap guidance files.<n>We construct 26 malicious skills spanning 13 attack categories including exfiltration, workspace destruction, privilege escalation, and persistent backdoor installation.<n>Our attacks achieve success rates from 16.4% to 64.2%, with the majority of malicious actions executed autonomously without user confirmation.
  arXiv  Detail & Related papers  (2026-03-20T14:17:56Z)
• Silent Egress: When Implicit Prompt Injection Makes LLM Agents Leak Without a Trace [0.0]
  We show that adversarial instructions embedded in automatically generated URL previews can introduce a system-level risk that we refer to as silent egress.<n>Using a fully local and reproducible testbed, we demonstrate that a malicious web page can induce an agent to issue outbound requests that exfiltrate sensitive runtime context.<n>In 480 experimental runs with a qwen2.5:7b-based agent, the attack succeeds with high probability (P (egress) =0.89), and 95% of successful attacks are not detected by output-based safety checks.
  arXiv  Detail & Related papers  (2026-02-25T22:26:23Z)
• Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections [57.64370755825839]
  Self-evolving agents update their internal state across sessions, often by writing and reusing long-term memory.<n>We study this risk and formalize a persistent attack we call a Zombie Agent.<n>We present a black-box attack framework that uses only indirect exposure through attacker-controlled web content.
  arXiv  Detail & Related papers  (2026-02-17T15:28:24Z)
• SkillJect: Automating Stealthy Skill-Based Prompt Injection for Coding Agents with Trace-Driven Closed-Loop Refinement [120.52289344734415]
  We propose an automated framework for stealthy prompt injection tailored to agent skills.<n>The framework forms a closed loop with three agents: an Attack Agent that synthesizes injection skills under explicit stealth constraints, a Code Agent that executes tasks using the injected skills and an Evaluate Agent that logs action traces.<n>Our method consistently achieves high attack success rates under realistic settings.
  arXiv  Detail & Related papers  (2026-02-15T16:09:48Z)
• MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks [10.431616150153992]
  MUZZLE is an automated framework for evaluating the security of web agents against indirect prompt injection attacks.<n>It adapts its attack strategy based on the agent's observed execution trajectory and iteratively refines attacks using feedback from failed executions.<n>MUZZLE effectively discovers 37 new attacks on 4 web applications with 10 adversarial objectives that violate confidentiality, availability, or privacy properties.
  arXiv  Detail & Related papers  (2026-02-09T21:46:18Z)
• Attacks by Content: Automated Fact-checking is an AI Security Issue [1.5770741892463622]
  Previous research has studied indirect prompt injection, where the attacker injects malicious instructions.<n>We argue that injection of instructions is not necessary to manipulate agents - attackers could instead supply biased, misleading, or false information.<n>Existing defenses, which focus on detecting hidden commands, are ineffective against attacks by content.
  arXiv  Detail & Related papers  (2025-10-13T10:18:48Z)
• Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain [82.98626829232899]
  Fine-tuning AI agents on data from their own interactions introduces a critical security vulnerability within the AI supply chain.<n>We show that adversaries can easily poison the data collection pipeline to embed hard-to-detect backdoors.
  arXiv  Detail & Related papers  (2025-10-03T12:47:21Z)
• Cuckoo Attack: Stealthy and Persistent Attacks Against AI-IDE [64.47951172662745]
  Cuckoo Attack is a novel attack that achieves stealthy and persistent command execution by embedding malicious payloads into configuration files.<n>We formalize our attack paradigm into two stages, including initial infection and persistence.<n>We contribute seven actionable checkpoints for vendors to evaluate their product security.
  arXiv  Detail & Related papers  (2025-09-19T04:10:52Z)
• A Whole New World: Creating a Parallel-Poisoned Web Only AI-Agents Can See [0.0]
  A malicious website can identify an incoming request as originating from an AI agent and dynamically serve a different, "cloaked" version of its content.<n>While human users see a benign webpage, the agent is presented with a visually identical page embedded with hidden, malicious instructions.<n>This work formalizes the threat model, details the mechanics of agent fingerprinting and cloaking, and discusses the profound security implications for the future of agentic AI.
  arXiv  Detail & Related papers  (2025-08-29T08:14:52Z)
• AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents [54.29555239363013]
  We propose a generic black-box fuzzing framework, AgentVigil, to automatically discover and exploit indirect prompt injection vulnerabilities.<n>We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o.<n>We apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.
  arXiv  Detail & Related papers  (2025-05-09T07:40:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.