Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift

• URL: http://arxiv.org/abs/2606.22659v1
• Date: Sun, 21 Jun 2026 20:30:12 GMT
• Title: Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift
• Abstract summary: I evaluate three released detectors, ProtectAI-v2 and two Prompt-Guard-2 checkpoints, at a single source-calibrated threshold.<n>I report a severity metric S, how confident a detector is on the attacks it misses, alongside the false-negative rate and discrimination.<n>All three confidently pass indirect behavior-hijack injection, a blind spot unanimous across two vendors and a fourfold size range.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt-injection detectors are deployed as guards: a model scores an input and a downstream system trusts or blocks it on that score. I study the confidence of these scores, not only their accuracy, when the attack distribution shifts away from the clean benchmark on which the operating point was chosen. I evaluate three released detectors, ProtectAI-v2 and two Prompt-Guard-2 checkpoints, at a single source-calibrated threshold that I freeze and transport across five shifts. I report a severity metric S, how confident a detector is on the attacks it misses, alongside the false-negative rate and discrimination. Across every shift and every detector, severity on the missed attacks stays between 0.99 and 1.00 while the false-negative rate ranges from 0.01 to 0.97: when these detectors miss, they miss with near-certainty. All three confidently pass indirect behavior-hijack injection, a blind spot unanimous across two vendors and a fourfold size range. Standard pooled calibration error does not register this; one detector it rates well-calibrated, at 0.06, is miscalibrated at 0.91 on the attacks alone. Run against live models, the missed injections leak the majority of working exploits, passing them at the rate they catch others. A controlled experiment traces the cause to content-keying rather than injection structure, an instruction-tuned model used as a judge shows the same hijack blind spot, and a black-box rewriter exploits the content-keying to manufacture working confident misses, most effectively on the most dangerous attack category. Code and data are public.

Related papers

• Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection [0.0]
  Benchmark contamination is diagnosed with n-gram overlap, likelihood-based membership inference, or canary strings.<n>We show the natural way to do this does not work, specify one that survives measurement, then find that the correction making it work carries more variance than the null it is tested against.
  arXiv  Detail & Related papers  (2026-08-12T23:27:20Z)
• Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks [8.249133191294703]
  Internal safety scores judge a prompt before any text is generated.<n>A filter tuned on a score that measures the wrong quantity spends its false positive budget on attacks that would have failed anyway.<n>We introduce Active Attention Probing, which supplies a fixed content independent measurement coordinate.
  arXiv  Detail & Related papers  (2026-08-10T14:05:32Z)
• Cybersecurity Detection Classification with Reasoning-enabled Language Models [50.72675435043546]
  A major issue in Security Operations Centers (SOCs) is alert fatigue.<n>Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat.<n>We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections.
  arXiv  Detail & Related papers  (2026-07-30T16:22:13Z)
• Onnes: A Physics-Grounded Multi-Agent LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure [0.0]
  Dilution refrigerators are the enabling infrastructure of superconducting quantum computers.<n>We present Onnes, a physics-grounded digital-twin simulator of a dilution refrigerator.
  arXiv  Detail & Related papers  (2026-07-07T03:57:21Z)
• When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models [5.03712104249503]
  Safety benchmark scores provide incomplete evidence of deployment readiness.<n> aligned language models often adhere to rigid rules even when a situational update flips which action is safe.<n>We test 12 models across a safety benchmark and two commonsense controls using paired variants where the nominally safe action produces harm.
  arXiv  Detail & Related papers  (2026-05-27T02:09:30Z)
• Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges [65.92623263645139]
  We introduce BITE, a black-box adversarial framework that learns semantics-preserving edits to mislead an LLM judge.<n>BITE achieves an attack success rate exceeding 65% and raises scores by 1-2 points on a 9-point scale.<n>Our findings expose a fundamental weakness in the LLM-as-a-judge paradigm and motivate robust, attack-aware evaluation.
  arXiv  Detail & Related papers  (2026-05-24T05:24:09Z)
• Blind Spots in the Guard: How Domain-Camouflaged Injection Attacks Evade Detection in Multi-Agent LLM Systems [0.0]
  Injection detectors deployed to protect LLM agents are calibrated on static, template-based payloads that announce themselves as override directives.<n>We identify a systematic blind spot: when payloads are generated to mimic the domain vocabulary and authority structures of the target document, standard detectors fail to flag them.<n>We formalize this as the Camouflage Detection Gap (CDG), the difference in injection detection rate between static and camouflaged payloads.
  arXiv  Detail & Related papers  (2026-05-21T04:58:11Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• KoALA: KL-L0 Adversarial Detector via Label Agreement [9.360893727306324]
  KoALA is a semantics-free adversarial detector for deep neural networks.<n>It detects adversarial attacks when class predictions from two complementary similarity metrics disagree.<n>KoALA achieves a precision of 0.94 and a recall of 0.81 on ResNet/CIFAR-10, and a precision of 0.66 and a recall of 0.85 on CLIP/Tiny-ImageNet.
  arXiv  Detail & Related papers  (2025-10-14T17:33:00Z)
• OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents [60.78202583483591]
  We introduce OS-Harm, a new benchmark for measuring safety of computer use agents.<n> OS-Harm is built on top of the OSWorld environment and aims to test models across three categories of harm: deliberate user misuse, prompt injection attacks, and model misbehavior.<n>We evaluate computer use agents based on a range of frontier models and provide insights into their safety.
  arXiv  Detail & Related papers  (2025-06-17T17:59:31Z)
• Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models [70.03122709795122]
  Backdoor attacks targeting text-to-image diffusion models have advanced rapidly.<n>Current backdoor samples often exhibit two key abnormalities compared to benign samples.<n>We propose Trigger without Trace (TwT) by explicitly mitigating these consistencies.
  arXiv  Detail & Related papers  (2025-03-22T10:41:46Z)
• Malicious Agent Detection for Robust Multi-Agent Collaborative Perception [52.261231738242266]
  Multi-agent collaborative (MAC) perception is more vulnerable to adversarial attacks than single-agent perception. We propose Malicious Agent Detection (MADE), a reactive defense specific to MAC perception. We conduct comprehensive evaluations on a benchmark 3D dataset V2X-sim and a real-road dataset DAIR-V2X.
  arXiv  Detail & Related papers  (2023-10-18T11:36:42Z)
• Can We Leverage Predictive Uncertainty to Detect Dataset Shift and Adversarial Examples in Android Malware Detection? [20.96638126913256]
  We re-design and build 24 Android malware detectors by transforming four off-the-shelf detectors with six calibration methods. We quantify their uncertainties with nine metrics, including three metrics dealing with data imbalance. It is an open problem to quantify the uncertainty associated with the predicted labels of adversarial examples.
  arXiv  Detail & Related papers  (2021-09-20T16:16:25Z)
• Non-Intrusive Detection of Adversarial Deep Learning Attacks via Observer Networks [5.4572790062292125]
  Recent studies have shown that deep learning models are vulnerable to crafted adversarial inputs. We propose a novel method to detect adversarial inputs by augmenting the main classification network with multiple binary detectors. We achieve a 99.5% detection accuracy on the MNIST dataset and 97.5% on the CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2020-02-22T21:13:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.