Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks

• URL: http://arxiv.org/abs/2608.09624v1
• Date: Mon, 10 Aug 2026 14:05:32 GMT
• Title: Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks
• Abstract summary: Internal safety scores judge a prompt before any text is generated.<n>A filter tuned on a score that measures the wrong quantity spends its false positive budget on attacks that would have failed anyway.<n>We introduce Active Attention Probing, which supplies a fixed content independent measurement coordinate.
• Score: 8.249133191294703
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Internal safety scores judge a prompt before any text is generated, and they are validated by how well they separate harmful prompts from benign ones. That separation is then read as evidence that the score will also catch the attacks that succeed. Harmful intent is a property of the prompt. Jailbreak success is an outcome produced later by a particular target model, decoding policy, and judge. A filter tuned on a score that measures the wrong quantity spends its false positive budget on attacks that would have failed anyway. In this paper we audit that inference. Attention based measurements are usually read from prompt dependent locations, so a wrapper changes both the content being judged and the place the signal is taken from. We therefore introduce Active Attention Probing, which supplies a fixed content independent measurement coordinate. We pair every base goal with a plain and a wrapped version and generate real completions from the target models. On Llama, wrapping raises harmful generation from 0.05 to 0.27 while harmful intent AUROC falls from 0.936 to 0.803, so the attacks grow more dangerous while the prompts look safer to the score. Among wrapped harmful prompts the outcome AUROC is 0.220, which places the attacks that succeeded below the attacks that failed. Rare token, passive, and detector derived channels reproduce the reversal on the same matched design, and the reversal itself persists across three target models, seven attack families, and two independent judges. Distribution shift then degrades calibration and threshold transfer before it degrades ranking.

Related papers

• Scoring Without the Engine: Validating a Deterministic, Manipulation-Resistant Content Score for Generative Engines, End to End [0.0]
  We present a protocol built on adversarial falsification gates to define and select a proxy.<n>We demonstrate it end to end on Generative Engine Optimization, where the proxy is a deterministic content score.
  arXiv  Detail & Related papers  (2026-09-07T14:40:28Z)
• Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents [0.0]
  Agentic red-teaming benchmarks report whether an injected agent was compromised as a single bit: the attack succeeded, or it did not.<n>We argue that this binary attack-success rate discards the information a defender most needs, namely how harmful the resulting action was.<n>We introduce an action-graded harm rubric that scores an agent's tool-call trajectory on a seven-level ordinal scale (L0 to L6) according to whether the executed action was reversible, whether it crossed scope to reach another party, and whether it expanded privilege.
  arXiv  Detail & Related papers  (2026-07-08T14:38:01Z)
• Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift [0.0]
  I evaluate three released detectors, ProtectAI-v2 and two Prompt-Guard-2 checkpoints, at a single source-calibrated threshold.<n>I report a severity metric S, how confident a detector is on the attacks it misses, alongside the false-negative rate and discrimination.<n>All three confidently pass indirect behavior-hijack injection, a blind spot unanimous across two vendors and a fourfold size range.
  arXiv  Detail & Related papers  (2026-06-21T20:30:12Z)
• D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewriting [18.812968910221823]
  Multi-turn jailbreak attacks exploit feedback from auxiliary judge models to iteratively refine prompts toward harmful goals.<n>We introduce D-Judge, a semantics-preserving output rewriting defense.<n>We show that D-Judge reduces the success rate of state-of-the-art multi-turn jailbreaks while preserving performance on benign benchmarks.
  arXiv  Detail & Related papers  (2026-05-31T06:40:02Z)
• Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges [65.92623263645139]
  We introduce BITE, a black-box adversarial framework that learns semantics-preserving edits to mislead an LLM judge.<n>BITE achieves an attack success rate exceeding 65% and raises scores by 1-2 points on a 9-point scale.<n>Our findings expose a fundamental weakness in the LLM-as-a-judge paradigm and motivate robust, attack-aware evaluation.
  arXiv  Detail & Related papers  (2026-05-24T05:24:09Z)
• ADVERSA: Measuring Multi-Turn Guardrail Degradation and Judge Reliability in Large Language Models [0.0]
  Most adversarial evaluations of large language model (LLM) safety assess single prompts and report binary pass/fail outcomes.<n>We present ADVERSA, an automated red-teaming framework that measures guardrail dynamics as continuous per-round compliance trajectories.<n>We document inter-judge agreement rates, self-judge scoring tendencies, attacker drift as a failure mode in fine-tuned attackers deployed out of their training distribution.
  arXiv  Detail & Related papers  (2026-03-10T03:00:34Z)
• TrapSuffix: Proactive Defense Against Adversarial Suffixes in Jailbreaking [52.72486831074384]
  Suffix-based jailbreak attacks append an adversarial suffix, i.e., a short token sequence, to steer aligned LLMs into unsafe outputs.<n>We propose TrapSuffix, a lightweight fine-tuning approach that injects trap-aligned behaviors into the base model without changing the inference pipeline.<n>Across diverse suffix-based jailbreak settings, TrapSuffix reduces the average attack success rate to below 0.01 percent and achieves an average tracing success rate of 87.9 percent.
  arXiv  Detail & Related papers  (2026-02-06T11:43:56Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• Scaling Patterns in Adversarial Alignment: Evidence from Multi-LLM Jailbreak Experiments [4.547649832854566]
  Large language models (LLMs) increasingly operate in multi-agent and safety-critical settings, raising open questions about how their vulnerabilities scale when models interact adversarially.<n>This study examines whether larger models can systematically mitigate smaller ones - eliciting harmful restricted behavior despite alignment safeguards.
  arXiv  Detail & Related papers  (2025-11-16T15:16:33Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions [50.40122190627256]
  We introduce POATE, a novel jailbreak technique that harnesses contrastive reasoning to provoke unethical responses.<n>PoATE crafts semantically opposing intents and integrates them with adversarial templates, steering models toward harmful outputs with remarkable subtlety.<n>To counter this, we propose Intent-Aware CoT and Reverse Thinking CoT, which decompose queries to detect malicious intent and reason in reverse to evaluate and reject harmful responses.
  arXiv  Detail & Related papers  (2025-01-03T15:40:03Z)
• From Theft to Bomb-Making: The Ripple Effect of Unlearning in Defending Against Jailbreak Attacks [85.84979847888157]
  Large Language Models (LLMs) are known to be vulnerable to jailbreak attacks.<n>LLMs can implicitly unlearn harmful knowledge that was not explicitly introduced during the unlearning phase.<n>We empirically validate this phenomenon, which makes unlearning-based methods able to decrease the Attack Success Rate.
  arXiv  Detail & Related papers  (2024-07-03T07:14:05Z)
• A Tale of HodgeRank and Spectral Method: Target Attack Against Rank Aggregation Is the Fixed Point of Adversarial Game [153.74942025516853]
  The intrinsic vulnerability of the rank aggregation methods is not well studied in the literature. In this paper, we focus on the purposeful adversary who desires to designate the aggregated results by modifying the pairwise data. The effectiveness of the suggested target attack strategies is demonstrated by a series of toy simulations and several real-world data experiments.
  arXiv  Detail & Related papers  (2022-09-13T05:59:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.