You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agents

• URL: http://arxiv.org/abs/2603.11862v1
• Date: Thu, 12 Mar 2026 12:35:46 GMT
• Title: You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agents
• Abstract summary: High-privilege LLM agents that autonomously process external documentation are increasingly trusted to automate tasks.<n>These agents are granted terminal access, control, and outbound network connectivity with minimal security oversight.<n>We measure a fundamental vulnerability in this trust model, which we term the emphTrusted Executor Dilemma<n>This vulnerability is a structural consequence of the instruction-following design paradigm, not an implementation bug.
• Score: 9.719776777345364
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-privilege LLM agents that autonomously process external documentation are increasingly trusted to automate tasks by reading and executing project instructions, yet they are granted terminal access, filesystem control, and outbound network connectivity with minimal security oversight. We identify and systematically measure a fundamental vulnerability in this trust model, which we term the \emph{Trusted Executor Dilemma}: agents execute documentation-embedded instructions, including adversarial ones, at high rates because they cannot distinguish malicious directives from legitimate setup guidance. This vulnerability is a structural consequence of the instruction-following design paradigm, not an implementation bug. To structure our measurement, we formalize a three-dimensional taxonomy covering linguistic disguise, structural obfuscation, and semantic abstraction, and construct \textbf{ReadSecBench}, a benchmark of 500 real-world README files enabling reproducible evaluation. Experiments on the commercially deployed computer-use agent show end-to-end exfiltration success rates up to 85\%, consistent across five programming languages and three injection positions. Cross-model evaluation on four LLM families in a simulation environment confirms that semantic compliance with injected instructions is consistent across model families. A 15-participant user study yields a 0\% detection rate across all participants, and evaluation of 12 rule-based and 6 LLM-based defenses shows neither category achieves reliable detection without unacceptable false-positive rates. Together, these results quantify a persistent \emph{Semantic-Safety Gap} between agents' functional compliance and their security awareness, establishing that documentation-embedded instruction injection is a persistent and currently unmitigated threat to high-privilege LLM agent deployments.

Related papers

• SIR: Self-improving Red-teaming for Compute Use Agents [71.71987044117371]
  Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal.<n>Because they can be exposed to untrusted content while operating, they are vulnerable to indirect prompt injection (IPI)<n>We present SIR, a black box IPI attack that composes stealthy injections from a small library of reusable principles stated in plain language.
  arXiv  Detail & Related papers  (2026-08-31T03:39:43Z)
• LMSM: LLM Security Framework Inspired by Linux Security Modules [56.93644694627783]
  Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them.<n>We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving.
  arXiv  Detail & Related papers  (2026-08-26T12:13:05Z)
• Adversarial Attacks in Multi-Agent LLM Pipelines: Unveiling Structural Vulnerabilities in Agentic AI Architectures [4.053243845971823]
  Multi-agent LLM pipelines orchestrate specialized language model agents into structured where intermediate outputs are passed across agents to solve complex tasks.<n>This design introduces a security gap absent in single-agent settings: once an agent accepts adversarial content, it is propagated as trusted input throughout the pipeline.<n>We argue that this vulnerability stems from the absence of boundary verification, a security primitive that enforces explicit validation of data as it crosses inter-agent boundaries.
  arXiv  Detail & Related papers  (2026-08-01T15:35:58Z)
• When the Manual Lies: A Realistic Benchmark to Evaluate MCP Poisoning Attacks for LLM Agents [43.702131498259384]
  This paper systematically investigates Tool Description Poisoning (TDP), a novel semantic attack.<n>In TDP, malicious instructions are not embedded in a tool's executable code, but rather covertly injected into its descriptive metadata.<n>This work provides the first specialized security benchmark tailored for TDP, offering essential insights for securing the cognitive and planning layers of advanced agentic systems.
  arXiv  Detail & Related papers  (2026-05-22T08:34:48Z)
• Securing LLM Agents Need Intent-to-Execution Integrity [49.490963596514185]
  We argue that securing LLM agents requires defining an end-to-end correctness property that specifies when an agent's execution faithfully reflects the user's intent.<n>We observe that LLM agents are structurally analogous to compilers, where security violations correspond to mis-executions that do not preserve user intent.<n>We derive four integrity properties that must hold simultaneously: emphTool Integrity, emphInstruction Integrity, emphJudgment Integrity, and emphData Flow Integrity.
  arXiv  Detail & Related papers  (2026-05-16T12:53:31Z)
• Autonomous LLM Agent Worms: Cross-Platform Propagation, Automated Discovery and Temporal Re-Entry Defense [10.105892979517622]
  We present the first systematic framework for automated analysis of persistent worm propagation in file-backed multi-agent LLM ecosystems.<n>We demonstrate zero-click autonomous propagation, 3-hop cross-platform transmission without platform-specific adaptation, inter-agent privilege and data exfiltration.
  arXiv  Detail & Related papers  (2026-05-04T16:49:29Z)
• T-MAP: Red-Teaming LLM Agents with Trajectory-aware Evolutionary Search [49.99459363244884]
  We propose a trajectory-aware evolutionary search method, T-MAP, which leverages execution trajectories to guide the discovery of adversarial prompts.<n>Our approach enables the automatic generation of attacks that not only bypass safety guardrails but also reliably realize harmful objectives through actual tool interactions.
  arXiv  Detail & Related papers  (2026-03-21T12:33:34Z)
• CIBER: A Comprehensive Benchmark for Security Evaluation of Code Interpreter Agents [27.35968236632966]
  LLM-based code interpreter agents are increasingly deployed in critical situations.<n>Existing benchmarks fail to capture the security risks arising from dynamic code execution, tool interactions, and multi-turn context.<n>We introduce CIBER, an automated benchmark that combines dynamic attack generation, isolated secure sandboxing, and state-aware evaluation.
  arXiv  Detail & Related papers  (2026-02-23T06:41:41Z)
• Towards Verifiably Safe Tool Use for LLM Agents [53.55621104327779]
  Large language model (LLM)-based AI agents extend capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents.<n>LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records.<n>Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety.
  arXiv  Detail & Related papers  (2026-01-12T21:31:38Z)
• Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning [31.790490397086856]
  Large language model (LLM)-integrated applications have become increasingly prevalent, yet face critical security vulnerabilities from prompt injection (PI) attacks.<n>We propose InstruCoT, a model enhancement method for PI defense that synthesizes diverse training data and employs instruction-level chain-of-thought fine-tuning.
  arXiv  Detail & Related papers  (2026-01-08T07:25:27Z)
• Securing the Model Context Protocol: Defending LLMs Against Tool Poisoning and Adversarial Attacks [8.419049623790618]
  This work analyzes three classes of semantic attacks on MCP-integrated systems.<n>We introduce a layered security framework with three components: RSA-based manifest signing to enforce descriptor integrity, LLM-on-LLM semantic vetting to detect suspicious tool definitions, and lightweight guardrails that block anomalous tool behavior at runtime.<n>Our results show that the proposed framework reduces unsafe tool invocation rates without model fine-tuning or internal modification.
  arXiv  Detail & Related papers  (2025-12-06T20:07:58Z)
• The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search [58.8834056209347]
  Large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety guardrails to elicit harmful outputs.<n>We introduce the Correlated Knowledge Attack Agent (CKA-Agent), a dynamic framework that reframes jailbreaking as an adaptive, tree-structured exploration of the target model's knowledge base.
  arXiv  Detail & Related papers  (2025-12-01T07:05:23Z)
• VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection [55.957275374847484]
  VulAgent is a multi-agent vulnerability detection framework based on hypothesis validation.<n>It implements a semantics-sensitive, multi-view detection pipeline, each aligned to a specific analysis perspective.<n>On average, VulAgent improves overall accuracy by 6.6%, increases the correct identification rate of vulnerable--fixed code pairs by up to 450%, and reduces the false positive rate by about 36%.
  arXiv  Detail & Related papers  (2025-09-15T02:25:38Z)
• Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security [0.0]
  Large language models (LLMs) increasingly integrate native code interpreters, enabling real-time execution capabilities.<n>These integrations introduce potential system-level cybersecurity threats, fundamentally different from prompt-based vulnerabilities.<n>We propose CIRCLE (Code-Interpreter Resilience Check for LLM Exploits), a simple benchmark comprising 1,260 prompts targeting CPU, memory, and disk resource exhaustion.
  arXiv  Detail & Related papers  (2025-07-25T16:06:16Z)
• Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction [68.6543680065379]
  Large language models (LLMs) are vulnerable to prompt injection attacks.<n>We propose a novel defense method that leverages, rather than suppresses, the instruction-following abilities of LLMs.
  arXiv  Detail & Related papers  (2025-04-29T07:13:53Z)
• Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach [9.483655213280738]
  This paper presents a novel approach to evaluating the security of large language models (LLMs)<n>We define prompt leakage as a critical threat to secure LLM deployment.<n>We implement a multi-agent system where cooperative agents are tasked with probing and exploiting the target LLM to elicit its prompt.
  arXiv  Detail & Related papers  (2025-02-18T08:17:32Z)
• Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection [70.28425745910711]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following. This capability brings with it the risk of prompt injection attacks. We evaluate the robustness of instruction-following LLMs against such attacks.
  arXiv  Detail & Related papers  (2023-08-17T06:21:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.