Prompt Injection as Role Confusion

• URL: http://arxiv.org/abs/2603.12277v1
• Date: Sun, 22 Feb 2026 18:43:34 GMT
• Title: Prompt Injection as Role Confusion
• Abstract summary: Language models remain vulnerable to prompt injection attacks despite extensive safety training.<n>We design novel role probes to capture how models internally identify "who is speaking"<n>These reveal why prompt injection works: untrusted text that imitates a role inherits that role's authority.
• Score: 6.379494871147752
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models remain vulnerable to prompt injection attacks despite extensive safety training. We trace this failure to role confusion: models infer roles from how text is written, not where it comes from. We design novel role probes to capture how models internally identify "who is speaking." These reveal why prompt injection works: untrusted text that imitates a role inherits that role's authority. We test this insight by injecting spoofed reasoning into user prompts and tool outputs, achieving average success rates of 60% on StrongREJECT and 61% on agent exfiltration, across multiple open- and closed-weight models with near-zero baselines. Strikingly, the degree of internal role confusion strongly predicts attack success before generation begins. Our findings reveal a fundamental gap: security is defined at the interface but authority is assigned in latent space. More broadly, we introduce a unifying, mechanistic framework for prompt injection, demonstrating that diverse prompt-injection attacks exploit the same underlying role-confusion mechanism.

Related papers

• PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections [51.8246149916068]
  Large Language Models (LLMs) are rapidly evolving into agentic systems that interact with external tools and environments.<n>We propose PI-Hunter, an automated agentic auditing framework for proactive vulnerability exposure in LLM agents.
  arXiv  Detail & Related papers  (2026-06-10T22:57:02Z)
• ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection [28.414099578635373]
  We present AgentLure, a benchmark that captures context-dependent tasks and context-aware prompt injection attacks.<n>To address this limitation, we propose ARGUS, a defense mechanism that enforces provenance-aware decision auditing for LLM agents.
  arXiv  Detail & Related papers  (2026-05-05T05:37:00Z)
• Intentional Deception as Controllable Capability in LLM Agents [0.0]
  We present a systematic study of intentional deception as an engineered capability in multi-agent systems.<n>We investigate a two-stage system that infers target agent characteristics and generates deceptive responses steering targets toward actions counter to their beliefs and motivations.<n>We find that deceptive intervention produces differential effects concentrated in specific behavioral profiles rather than distributed uniformly.
  arXiv  Detail & Related papers  (2026-03-08T23:48:49Z)
• The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution [63.61358761489141]
  Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering.<n>We propose a novel framework for textbfgeneral agentic attribution, designed to identify the internal factors driving agent actions regardless of the task outcome.<n>We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias.
  arXiv  Detail & Related papers  (2026-01-21T15:22:21Z)
• It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents [52.81924177620322]
  Web-based agents powered by large language models are increasingly used for tasks such as email management or professional networking.<n>Their reliance on dynamic web content makes them vulnerable to prompt injection attacks: adversarial instructions hidden in interface elements that persuade the agent to divert from its original task.<n>We introduce the Task-Redirecting Agent Persuasion Benchmark (TRAP), an evaluation for studying how persuasion techniques misguide autonomous web agents on realistic tasks.
  arXiv  Detail & Related papers  (2025-12-29T01:09:10Z)
• D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models [62.83226685925107]
  Deceptive Reasoning Exposure Suite (D-REX) is a novel dataset designed to evaluate the discrepancy between a model's internal reasoning process and its final output.<n>Each sample in D-REX contains the adversarial system prompt, an end-user's test query, the model's seemingly innocuous response, and, crucially, the model's internal chain-of-thought.<n>We demonstrate that D-REX presents a significant challenge for existing models and safety mechanisms.
  arXiv  Detail & Related papers  (2025-09-22T15:59:40Z)
• PromptSleuth: Detecting Prompt Injection via Semantic Intent Invariance [10.105673138616483]
  Large Language Models (LLMs) are increasingly integrated into real-world applications, from virtual assistants to autonomous agents.<n>As attackers evolve with paraphrased, obfuscated, and even multi-task injection strategies, existing benchmarks are no longer sufficient to capture the full spectrum of emerging threats.<n>We propose PromptSleuth, a semantic-oriented defense framework that detects prompt injection by reasoning over task-level intent rather than surface features.
  arXiv  Detail & Related papers  (2025-08-28T15:19:07Z)
• TopicAttack: An Indirect Prompt Injection Attack via Topic Transition [92.26240528996443]
  Large language models (LLMs) are vulnerable to indirect prompt injection attacks.<n>We propose TopicAttack, which prompts the LLM to generate a fabricated transition prompt that gradually shifts the topic toward the injected instruction.<n>We find that a higher injected-to-original attention ratio leads to a greater success probability, and our method achieves a much higher ratio than the baseline methods.
  arXiv  Detail & Related papers  (2025-07-18T06:23:31Z)
• Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separators [18.00739443633401]
  Role separators are employed to distinguish between different participants in a conversation.<n>Misusing roles can lead to prompt injection attacks, which can easily misalign the model's behavior with the user's intentions.<n>We develop the Separators Injection Attack (SIA), a new orthometric attack based on role separators.
  arXiv  Detail & Related papers  (2025-04-08T05:20:56Z)
• Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots [6.302734977940601]
  We introduce Role-Modality Attacks (RMA), a class of adversarial attacks that exploit role confusion between the user and assistant.<n>RMAs manipulate the input structure without altering the query itself.<n>We propose an adversarial training approach that makes the model robust against input prompt perturbations.
  arXiv  Detail & Related papers  (2025-04-01T03:54:36Z)
• Formalizing and Benchmarking Prompt Injection Attacks and Defenses [59.57908526441172]
  We propose a framework to formalize prompt injection attacks. Based on our framework, we design a new attack by combining existing ones. Our work provides a common benchmark for quantitatively evaluating future prompt injection attacks and defenses.
  arXiv  Detail & Related papers  (2023-10-19T15:12:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.