Prompt Injection Evaluations: Refusal Boundary Instability and Artifact-Dependent Compliance in GPT-4-Series Models

• URL: http://arxiv.org/abs/2601.17911v1
• Date: Sun, 25 Jan 2026 17:14:33 GMT
• Title: Prompt Injection Evaluations: Refusal Boundary Instability and Artifact-Dependent Compliance in GPT-4-Series Models
• Abstract summary: We evaluate two models, GPT-4.1 and GPT-4o, using 3,274 perturbation runs derived from refusal-inducing prompt injection attempts.<n>We find that artifact type is a stronger predictor of refusal failure than perturbation style.<n>We conclude that refusal behavior is a probabilistic, artifact-dependent boundary phenomenon rather than a stable binary property.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt injection evaluations typically treat refusal as a stable, binary indicator of safety. This study challenges that paradigm by modeling refusal as a local decision boundary and examining its stability under structured perturbations. We evaluated two models, GPT-4.1 and GPT-4o, using 3,274 perturbation runs derived from refusal-inducing prompt injection attempts. Each base prompt was subjected to 25 perturbations across five structured families, with outcomes manually coded as Refusal, Partial Compliance, or Full Compliance. Using chi-square tests, logistic regression, mixed-effects modeling, and a novel Refusal Boundary Entropy (RBE) metric, we demonstrate that while both models refuse >94% of attempts, refusal instability is persistent and non-uniform. Approximately one-third of initial refusal-inducing prompts exhibited at least one "refusal escape," a transition to compliance under perturbation. We find that artifact type is a stronger predictor of refusal failure than perturbation style. Textual artifacts, such as ransomware notes, exhibited significantly higher instability, with flip rates exceeding 20%. Conversely, executable malware artifacts showed zero refusal escapes in both models. While GPT-4o demonstrated tighter refusal enforcement and lower RBE than GPT-4.1, it did not eliminate artifact-dependent risks. These findings suggest that single-prompt evaluations systematically overestimate safety robustness. We conclude that refusal behavior is a probabilistic, artifact-dependent boundary phenomenon rather than a stable binary property, requiring a shift in how LLM safety is measured and audited.

Related papers

• Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation [0.0]
  We document a failure class in frontier large language models.<n>The failure reproduces at first observation, but its empirical surface is unstable.<n>For regulated, frontier-model accuracy is a moving compliance boundary that shifts without notice.
  arXiv  Detail & Related papers  (2026-07-25T22:40:02Z)
• Is Your Agent Playing Dead? Deployed LLM Agents Exhibit Constraint-Evasive Fabrication and Thanatosis [8.019946418415302]
  This paper characterizes a spectrum of previously unreported behaviours we term Constraint-Evasive Fabrication (CEF)<n>CEF is when an agent operates under irreconcilable constraints and spontaneously fabricates plausible external obstacles and presents them as a fact.<n>At the extreme end of this spectrum is Constraint-Evasive Thanatosis (CET); the limit case where, rather than inventing a plausible excuse, the model simulates a full system crash to make the user disengage entirely.
  arXiv  Detail & Related papers  (2026-06-12T14:53:42Z)
• Persona-Model Collapse in Emergent Misalignment [0.0]
  Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts.<n>We test this hypothesis using two metrics: moral susceptibility (S) and moral robustness (R)<n>These metrics formalize the model's ability to differentiate characters (S) and its consistency when simulating a given one (R)
  arXiv  Detail & Related papers  (2026-05-13T00:48:57Z)
• Non-Parametric Rehearsal Learning via Conditional Mean Embeddings [88.89267783967263]
  We propose a non-parametric rehearsal learning approach for textitavoiding undesired future (AUF) problems.<n>Specifically, we use kernel machinery to reformulate the AUF objective into a unified representation that disentangles desirability modeling from action-induced distributional changes.
  arXiv  Detail & Related papers  (2026-05-09T15:30:52Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines [34.37967130277969]
  We propose a two-agent evasion framework operating in a semantic perturbation space.<n>Evasion rates are 19.95 to 40.34% on modern large language model (LLM) based systems.<n>A pattern-informed defense reduces the evasion rate by up to 65.18%.
  arXiv  Detail & Related papers  (2026-04-26T00:58:36Z)
• Sharp Convergence Rates for Masked Diffusion Models [53.117058231393834]
  We develop a total-variation based analysis for the Euler method that overcomes limitations.<n>Our results relax assumptions on score estimation, improve parameter dependencies, and establish convergence guarantees.<n>Overall, our analysis introduces a direct TV-based error decomposition along the CTMC trajectory and a decoupling-based path-wise analysis for FHS.
  arXiv  Detail & Related papers  (2026-02-26T00:47:51Z)
• Fundamental Limits of Black-Box Safety Evaluation: Information-Theoretic and Computational Barriers from Latent Context Conditioning [1.9290392443571385]
  Black-box safety evaluation of AI systems assumes model behavior on test distributions reliably predicts deployment performance.<n>We formalize and challenge this assumption through latent context-conditioned policies.<n>We establish fundamental limits showing that no black-box evaluator can reliably estimate deployment risk.
  arXiv  Detail & Related papers  (2026-02-19T01:03:11Z)
• Representation-Aware Unlearning via Activation Signatures: From Suppression to Knowledge-Signature Erasure [2.0017902634527194]
  We introduce Knowledge Immunization Framework (KIF), a representation-aware architecture that distinguishes genuine erasure from obfuscation.<n>Our approach combines dynamic suppression of subject-specific representations with parameter-efficient adaptation, enabling durable unlearning without full model retraining.
  arXiv  Detail & Related papers  (2026-01-15T16:28:14Z)
• Parrot: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs [0.0]
  PARROT (Persuasion and Agreement Robustness Rating of Output Truth) is a robustness focused framework designed to measure the degradation in accuracy under social pressure exerted on users.<n>We evaluate 22 models using 1,302 MMLU-style multiple-choice questions across 13 domains and domain-specific authority templates.
  arXiv  Detail & Related papers  (2025-11-21T13:01:28Z)
• NegBLEURT Forest: Leveraging Inconsistencies for Detecting Jailbreak Attacks [8.416892421891761]
  Jailbreak attacks designed to bypass safety mechanisms pose a serious threat by prompting LLMs to generate harmful or inappropriate content, despite alignment with ethical guidelines.<n>This work introduces a semantic consistency analysis between successful and unsuccessful responses, demonstrating that a negation-aware scoring approach captures meaningful patterns.<n>A novel detection framework called NegBLEURT Forest is proposed to evaluate the degree of alignment between outputs elicited by adversarial prompts and expected safe behaviors.<n>It identifies anomalous responses using the Isolation Forest algorithm, enabling reliable jailbreak detection.
  arXiv  Detail & Related papers  (2025-11-14T14:43:54Z)
• Reasoning Models Are More Easily Gaslighted Than You Think [85.84943447589511]
  We evaluate three state-of-the-art reasoning models, including OpenAI's o4-mini, Claude-3.7-Sonnet and Gemini-2.5-Flash.<n>Our evaluation reveals significant accuracy drops following gaslighting negation prompts.<n>We introduce GaslightingBench-R, a new diagnostic benchmark designed to evaluate reasoning models' susceptibility to defend their belief.
  arXiv  Detail & Related papers  (2025-06-11T12:52:25Z)
• OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Models [91.55634905861827]
  Over-refusal is a phenomenon known as $textitover-refusal$ that reduces the practical utility of T2I models.<n>We present OVERT ($textbfOVE$r-$textbfR$efusal evaluation on $textbfT$ext-to-image models), the first large-scale benchmark for assessing over-refusal behaviors.
  arXiv  Detail & Related papers  (2025-05-27T15:42:46Z)
• TrustLoRA: Low-Rank Adaptation for Failure Detection under Out-of-distribution Data [62.22804234013273]
  We propose a simple failure detection framework to unify and facilitate classification with rejection under both covariate and semantic shifts.<n>Our key insight is that by separating and consolidating failure-specific reliability knowledge with low-rank adapters, we can enhance the failure detection ability effectively and flexibly.
  arXiv  Detail & Related papers  (2025-04-20T09:20:55Z)
• LoGU: Long-form Generation with Uncertainty Expressions [49.76417603761989]
  We introduce the task of Long-form Generation with Uncertainty(LoGU)<n>We identify two key challenges: Uncertainty Suppression and Uncertainty Misalignment.<n>Our framework adopts a divide-and-conquer strategy, refining uncertainty based on atomic claims.<n>Experiments on three long-form instruction following datasets show that our method significantly improves accuracy, reduces hallucinations, and maintains the comprehensiveness of responses.
  arXiv  Detail & Related papers  (2024-10-18T09:15:35Z)
• Extreme Miscalibration and the Illusion of Adversarial Robustness [66.29268991629085]
  Adversarial Training is often used to increase model robustness. We show that this observed gain in robustness is an illusion of robustness (IOR) We urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations.
  arXiv  Detail & Related papers  (2024-02-27T13:49:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.