Related papers: PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift

PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift

URL: http://arxiv.org/abs/2609.15017v1
Date: Mon, 14 Sep 2026 04:28:55 GMT
Title: PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift
Abstract summary: We present PIDS-Bench, a benchmark that jointly evaluates attack detection and benign false-positive behavior at fixed thresholds.<n>We find that hard-negative augmentation nearly eliminates over-defense on curated stress inputs but leaves it substantially intact on externally-sourced prompts.
Score: 0.2291770711277359
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Prompt-injection detectors are typically evaluated using aggregate F1 on in-distribution test data, which offers limited insight into behavior under distribution shift, particularly on the benign side of the decision boundary, where false positives impose direct operational cost yet are seldom measured. We present PIDS-Bench, a frozen multi-axis benchmark that jointly evaluates attack detection and benign false-positive behavior at fixed thresholds, spanning in-distribution inputs, hard-benign prompts that mimic injection structure without malicious intent, obfuscated attacks, and domain and structural distribution shifts. We evaluate seven detectors (learned baselines, external prompt-injection classifiers, and broad-safety comparators) alongside a rule-based lower-bound reference. Multi-axis evaluation exposes a failure mode that aggregate F1 conceals. A detector exceeding F1 = 0.98 on the held-out split still misclassifies roughly one-third of an externally-sourced benign subset drawn from public corpora and restricted to security-adjacent content. Across a full threshold sweep and five training seeds, no internal detector reaches an operating point satisfying F1 >= 0.95 and hard-benign FPR <= 0.10 together on this stress distribution. Decomposing by provenance, we find that hard-negative augmentation nearly eliminates over-defense on curated stress inputs but leaves it substantially intact on externally-sourced prompts, a pattern we term provenance-sensitive over-defense. The asymmetry holds across both fine-tuned architectures and does not diminish as the augmentation pool grows, with the externally-sourced FPR remaining far above the 0.10 target. Whether augmentation matched to the externally-sourced distribution would close this gap is untested; threshold calibration and curated-style augmentation alone do not.

Related papers

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
arXiv Detail & Related papers (2026-06-26T17:59:15Z)
Interpretable Uncertainty Routing Separating Emotion Ambiguity from Distribution Shift in Facial Expression Recognition [0.0]
Uncertainty-Aware Routing (UAR) is an inference-time routing mechanism that exploits the separation between ambiguity and shift.<n>A proposed dual-validation protocol reveals that aleatoric recovers annotator disagreement with Spearman correlation.<n>UAR retains approximately 1.8 times more ambiguous in-distribution faces than single-uncertainty routing at a matched out-of-distribution rejection rate.
arXiv Detail & Related papers (2026-06-21T23:54:44Z)
Nous: An Attempt to Extract and Inject the Cognition Behind Prediction-Market Behavior [0.0]
We ask whether human cognitive diversity can be recovered from behavior and transferred to LLM agents.<n>Nous extracts a structured eight-dimension behavioral profile from real Polymarket trading activity and injects it into agents through prompts.<n>We position Nous as measuring the cognitive-monoculture problem and the limits of a prompt-level remedy.
arXiv Detail & Related papers (2026-06-11T08:18:25Z)
How Useful is Causal Invariance for Domain Adaptation in Finite-Sample Settings? [58.740078141879984]
Machine learning models often degrade when they are deployed on a target distribution that differs from the source distributions they were trained on.<n>Recent work in causality-based domain generalization has shown how shared causal structure can induce invariant predictors.<n>In this paper, we study when full or partial causal knowledge can provably improve supervised domain adaptation.
arXiv Detail & Related papers (2026-06-10T21:07:49Z)
The Distributed Detectability Band Against Marginal-Preserving Attacks [0.0]
AI-control monitors score individual agent actions to detect misbehavior.<n>Real harm can be distributed across many benign-looking steps, each individually below any per-step alarm.<n>We construct a marginal-preserving, correlation-encoded distributed-sabotage attack.
arXiv Detail & Related papers (2026-06-09T06:04:30Z)
Quantum Machine Learning for Cyber-Physical Anomaly Detection in Unmanned Aerial Vehicles: A Leakage-Free Evaluation with Proxy-Audited Feature Sets [0.0]
Unmanned aerial vehicles (UAVs) are cyber-physical systems whose attack surface spans networked avionics and on-board sensor fusion.<n>We present a leakage-free evaluation of quantum machine learning for UAV anomaly detection on the multi-sensor TLM:UAV benchmark.
arXiv Detail & Related papers (2026-05-19T01:05:16Z)
Improving Search Agent with One Line of Code [68.58667107354253]
Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
arXiv Detail & Related papers (2026-03-10T04:07:39Z)
Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
arXiv Detail & Related papers (2026-02-10T21:08:53Z)
Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlap [3.351714665243138]
We re-visit the task of off-policy evaluation in Markov decision processes (MDPs) under a weaker notion of distributional overlap.<n>We introduce a class of truncated doubly robust (TDR) estimators which we find to perform well in this setting.<n>We find that, in our experiments, appropriate truncation plays a major role in enabling accurate off-policy evaluation when strong distributional overlap does not hold.
arXiv Detail & Related papers (2024-02-13T03:55:56Z)
On the Practicality of Differential Privacy in Federated Learning by Tuning Iteration Times [51.61278695776151]
Federated Learning (FL) is well known for its privacy protection when training machine learning models among distributed clients collaboratively. Recent studies have pointed out that the naive FL is susceptible to gradient leakage attacks. Differential Privacy (DP) emerges as a promising countermeasure to defend against gradient leakage attacks.
arXiv Detail & Related papers (2021-01-11T19:43:12Z)
Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment [52.02794488304448]
We propose a new distribution alignment method based on a log-likelihood ratio statistic and normalizing flows. We experimentally verify that minimizing the resulting objective results in domain alignment that preserves the local structure of input domains.
arXiv Detail & Related papers (2020-03-26T22:10:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.