Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals

• URL: http://arxiv.org/abs/2605.26999v1
• Date: Tue, 26 May 2026 13:19:25 GMT
• Title: Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals
• Abstract summary: We present a deployment-aware evaluation of prompt injection detection using a multi-model and multi-regime experimental framework.<n>We compare lexical, semantic, structural, and transformer-based detectors across multiple out-of-distribution settings.<n>Results show that detection performance is highly regime-dependent and sensitive to threshold selection.
• Score: 11.542407696902115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt injection poses a critical threat to the safe deployment of large language models, yet existing detection approaches are typically evaluated under limited settings that do not reflect real-world operating constraints. In this work, we present a deployment-aware evaluation of prompt injection detection using a multi-model and multi-regime experimental framework. We compare lexical, semantic, structural, and transformer-based detectors across multiple out-of-distribution settings, repeated data splits, and both ranking and thresholded deployment metrics. We introduce interpretable structural signals that capture hierarchy overrides, system prompt spoofing, role redefinition, and evasion patterns, and assess their contribution both within sparse models and in combination with strong encoder baselines. Our results show that detection performance is highly regime-dependent and sensitive to threshold selection, with no single model dominating across all settings. Transformer-based models achieve the strongest overall performance, while structural signals provide modest but consistent gains in certain regimes and improve low false positive rate behaviour in harder scenarios. These findings highlight the gap between ranking performance and deployment effectiveness and underscore the importance of evaluating prompt injection defences under realistic operational constraints. Code will be released.

Related papers

• OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning [4.173831365316023]
  We present OOD-RL-Bench, a framework designed to evaluate OOD detectors against categories of anomalies injected into trajectories.<n> Detectors and anomaly injectors are integrated through shared interfaces and configuration.<n>We assess the performance of each detector across a suite of anomaly types using matched-time AUROC, matched-time AUPRC, matched-time false-positive rate, detection delay, and segmented-onset metrics.
  arXiv  Detail & Related papers  (2026-07-14T08:57:04Z)
• From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Models [51.470637128942336]
  We propose a factorised study of probe-based uncertainty estimation (UE) under matched conditions.<n>Our results show that raw hidden states and attention features are difficult to outperform in-domain.<n>We train benchmark-based pretrained probes that transfer reasonably well to open-ended factual generation.
  arXiv  Detail & Related papers  (2026-06-26T03:23:15Z)
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• Active Bayesian Inference for Robust Control under Sensor False Data Injection Attacks [10.312968200748115]
  We present a framework for bridging the gap between sensor attack detection and recovery in cyber-physical systems.<n>We propose a threshold-based probing strategy and show its effectiveness via a simplified partially observable Markov decision process (POMDP) formulation.
  arXiv  Detail & Related papers  (2026-04-13T12:54:24Z)
• AgentScore: Autoformulation of Deployable Clinical Scoring Systems [45.88028371034407]
  We introduce AgentScore, which performs semantically guided optimization in unit-weighted clinical checklists.<n>AgentScore outperforms existing score-generation methods and achieves AUC comparable to more flexible interpretable models.<n>On two additional externally validated tasks, AgentScore achieves higher discrimination than established guideline-based scores.
  arXiv  Detail & Related papers  (2026-01-29T21:11:06Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity [45.92643973404507]
  We investigate whether aligned models are vulnerable to Preference-Undermining Attacks (PUA), a class of manipulative prompting strategies.<n>Surprisingly, more advanced models are sometimes more susceptible to manipulative prompts.
  arXiv  Detail & Related papers  (2026-01-10T15:16:23Z)
• Cascading multi-agent anomaly detection in surveillance systems via vision-language models and embedding-based classification [0.0]
  This work introduces a cascading multi-agent framework that unifies complementary paradigms into a coherent and interpretable architecture.<n>Early modules perform reconstruction-gated filtering and object-level assessment, while higher-level reasoning agents are selectively invoked to interpret semantically ambiguous events.<n>The framework advances beyond conventional detection pipelines by combining early-exit efficiency, adaptive multi-agent reasoning, and explainable anomaly attribution, establishing a reproducible and energy-efficient foundation for scalable intelligent visual monitoring.
  arXiv  Detail & Related papers  (2026-01-08T11:31:47Z)
• A Few Large Shifts: Layer-Inconsistency Based Minimal Overhead Adversarial Example Detection [13.109309606764754]
  We introduce a plug-in detection framework that leverages internal layer-wise inconsistencies within the target model itself.<n>Our method achieves state-of-the-art detection performance with negligible computational overhead.
  arXiv  Detail & Related papers  (2025-05-19T00:48:53Z)
• Hierarchical Polysemantic Feature Embedding for Autonomous Ransomware Detection [0.0]
  The evolution of ransomware requires the development of more sophisticated detection techniques.<n>The proposed framework embeds ransomware-relevant features into a non-Euclidean space.<n> Experimental evaluations demonstrated that the framework consistently outperformed conventional machine learning-based models.<n>The proposed method maintains a balance between detection performance and processing overhead, making it a viable candidate for real-world cybersecurity applications.
  arXiv  Detail & Related papers  (2025-02-09T21:46:36Z)
• Auto-Prompt Generation is Not Robust: Prompt Optimization Driven by Pseudo Gradient [50.15090865963094]
  We introduce PertBench, a comprehensive benchmark dataset that includes a wide range of input perturbations.<n>Our analysis reveals substantial vulnerabilities in existing prompt generation strategies.<n>We propose PGO, a gradient-free prompt generation framework that leverages perturbation types as pseudo-gradient signals.
  arXiv  Detail & Related papers  (2024-12-24T06:05:08Z)
• Unsupervised Model Diagnosis [49.36194740479798]
  This paper proposes Unsupervised Model Diagnosis (UMO) to produce semantic counterfactual explanations without any user guidance. Our approach identifies and visualizes changes in semantics, and then matches these changes to attributes from wide-ranging text sources.
  arXiv  Detail & Related papers  (2024-10-08T17:59:03Z)
• Clustering Effect of (Linearized) Adversarial Robust Models [60.25668525218051]
  We propose a novel understanding of adversarial robustness and apply it on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy.
  arXiv  Detail & Related papers  (2021-11-25T05:51:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.