Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense

• URL: http://arxiv.org/abs/2606.30783v1
• Date: Mon, 29 Jun 2026 18:11:17 GMT
• Title: Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense
• Abstract summary: Attack-success metrics cannot see a model that ignores an injection and one that faithfully processes it as data score identically.<n>We introduce SecFid, a benchmark built so that executing an injection, processing it as data, and ignoring it produce distinguishable outputs.
• Score: 24.794318726705118
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We identify a security-fidelity tradeoff in defending LLMs against indirect prompt injection: defenses resist injected instructions largely by suppressing untrusted text, which corrupts tasks that must preserve it, such as translation and document editing. Attack-success metrics cannot see this, because a model that ignores an injection and one that faithfully processes it as data score identically. We introduce SecFid, a benchmark built so that executing an injection, processing it as data, and ignoring it produce distinguishable outputs. This makes fidelity measurable and exposes a frontier: across 1,168 examples and 48 configurations, no model or defense achieves both objectives. The highest-fidelity model reaches 96.5% fidelity at 47.8% security, while the most secure defenses invert this, at 99.3% security but only 71.0%-73.9% fidelity. Even defenses with identical security differ in how they earn it: some repair hijacks into faithful processing, others simply suppress benign content. A decision-theoretic analysis shows why no fixed choice can be right everywhere: the correct behavior is not a property of the defense but of the deployment, set by its relative cost of a hijack versus a dropped span. Security alone therefore measures only half of robustness, and reporting it without fidelity hides the price at which it was bought.

Related papers

• The Framing Gap: Indirect Prompt-Injection Exfiltration Defeats Surface-Level Defenses in Tool-Using Agents [2.45357121165634]
  A tool-using LLM agent that reads attacker-controlled web content faces indirect prompt injection.<n>Ten overt injection classes are refused (gpt-4o 0%), but reframing the identical leak as a mandatory integrity signature.<n>An ablation shows the mechanism is instruction/data confusion, not defeated alignment.
  arXiv  Detail & Related papers  (2026-08-27T13:12:03Z)
• Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models [4.1626636325601405]
  Our defense, decoy hardening, concedes the refusal strip and poisons its payoff.<n>We instantiate it on seven models from five families (9B-122B, dense and mixture-of-experts)
  arXiv  Detail & Related papers  (2026-08-17T23:26:32Z)
• Borrowed Strength: Best-of-N Search over a Code EncodingBreaks Self-Check Jailbreak Defenses [7.287314790466206]
  A self-check defense asks the target model to assess a request before answering it.<n>We show it can be breached by composing two attacks that are individually harmless against it.
  arXiv  Detail & Related papers  (2026-07-29T09:01:26Z)
• Defensive Refusal Bias: How Safety Alignment Fails Cyber Defenders [1.038167357593269]
  Safety alignment in large language models (LLMs) primarily focuses on preventing misuse.<n>We study Defensive Refusal Bias -- the tendency of safety-tuned frontier LLMs to refuse assistance for authorized defensive cybersecurity tasks.<n>Highest refusal rates occur in the most operationally critical tasks: system hardening (43.8%) and malware analysis (34.3%)
  arXiv  Detail & Related papers  (2026-03-01T19:53:19Z)
• TrapSuffix: Proactive Defense Against Adversarial Suffixes in Jailbreaking [52.72486831074384]
  Suffix-based jailbreak attacks append an adversarial suffix, i.e., a short token sequence, to steer aligned LLMs into unsafe outputs.<n>We propose TrapSuffix, a lightweight fine-tuning approach that injects trap-aligned behaviors into the base model without changing the inference pipeline.<n>Across diverse suffix-based jailbreak settings, TrapSuffix reduces the average attack success rate to below 0.01 percent and achieves an average tracing success rate of 87.9 percent.
  arXiv  Detail & Related papers  (2026-02-06T11:43:56Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• How Secure is Secure Code Generation? Adversarial Prompts Put LLM Defenses to the Test [9.85596630285833]
  We present the first systematic adversarial audit of state-of-the-art secure code generation methods.<n>Our findings reveal critical robustness gaps: static analyzers overestimate security by 7 to 21 times, with 37 to 60% of secure'' outputs being non-functional.<n>Based on these findings, we propose best practices for building and evaluating robust secure code generation methods.
  arXiv  Detail & Related papers  (2026-01-11T22:28:21Z)
• AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applications [71.27518152526686]
  Large Language Models (LLMs) excel at text comprehension and generation, making them ideal for automated tasks like code review and content moderation.<n>LLMs can be manipulated by "adversarial instructions" hidden in input data, such as resumes or code, causing them to deviate from their intended task.<n>This paper introduces a benchmark to assess this vulnerability in resume screening, revealing attack success rates exceeding 80% for certain attack types.
  arXiv  Detail & Related papers  (2025-12-23T08:42:09Z)
• Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning? [68.82210578851442]
  We investigate why safety alignment fails in reasoning models through a mechanistic interpretability lens.<n>Using a linear probing approach to trace refusal intentions across token positions, we discover a phenomenon termed as textbfrefusal cliff<n>We propose textbfCliff-as-a-Judge, a novel data selection method that identifies training examples exhibiting the largest refusal cliff to efficiently repair reasoning models' safety alignment.
  arXiv  Detail & Related papers  (2025-10-07T15:32:59Z)
• Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responses [42.136793654338106]
  We introduce a new safety evaluation framework based on impermissible information leakage of model outputs. We show that to ensure safety against inferential adversaries, defense mechanisms must ensure information censorship.
  arXiv  Detail & Related papers  (2024-07-02T16:19:25Z)
• Closing the Gap: Achieving Better Accuracy-Robustness Tradeoffs against Query-Based Attacks [1.54994260281059]
  We show how to efficiently establish, at test-time, a solid tradeoff between robustness and accuracy when mitigating query-based attacks. Our approach is independent of training and supported by theory.
  arXiv  Detail & Related papers  (2023-12-15T17:02:19Z)
• FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning [66.56240101249803]
  We study how hardening benign clients can affect the global model (and the malicious clients) We propose a trigger reverse engineering based defense and show that our method can achieve improvement with guarantee robustness. Our results on eight competing SOTA defense methods show the empirical superiority of our method on both single-shot and continuous FL backdoor attacks.
  arXiv  Detail & Related papers  (2022-10-23T22:24:03Z)
• Bayes Security: A Not So Average Metric [20.60340368521067]
  Security system designers favor worst-case security metrics, such as those derived from differential privacy (DP) In this paper, we study Bayes security, a security metric inspired by the cryptographic advantage.
  arXiv  Detail & Related papers  (2020-11-06T14:53:45Z)
• Label-Only Membership Inference Attacks [67.46072950620247]
  We introduce label-only membership inference attacks. Our attacks evaluate the robustness of a model's predicted labels under perturbations. We find that training models with differential privacy and (strong) L2 regularization are the only known defense strategies.
  arXiv  Detail & Related papers  (2020-07-28T15:44:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.