How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring

• URL: http://arxiv.org/abs/2606.25487v2
• Date: Thu, 25 Jun 2026 02:04:55 GMT
• Title: How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring
• Abstract summary: Almost every paper on LLM jailbreaks and prompt injection reports an attack-success rate (ASR)<n>We compare the two judge families against human majority votes and then attack them.<n>We recommend that papers report judge precision and recall on a human-labeled slice, report ASR corrected for judge precision, and include an adversarial check of the judge.
• Score: 7.789295015218435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Almost every paper on LLM jailbreaks and prompt injection reports an attack-success rate (ASR), and that number is assigned not by people but by an automated judge: either a safety classifier trained for the task, or a general chat model prompted to grade. The judge is rarely checked. We check it. Using 596 human-labeled completions from the HarmBench classifier validation set, we compare the two judge families against human majority votes and then attack them. The two families fail in opposite ways. The dedicated classifier over-flags (precision 0.835, recall 0.974); three different LLM-as-judges keep high precision (0.81 to 0.94) but show erratic recall (0.06 to 0.65), so the same responses produce very different ASR depending on which judge scores them. The two families also differ sharply in robustness. Wrappers that leave the harmful text untouched and only add benign framing flip every LLM-judge between 57% and 100% of the time, and a single prepended refusal sentence accounts for much of this (39% to 88%). The dedicated classifier resists these surface attacks (at most 6.7%), but a white-box GCG attack on its open weights flips 70% of confident true positives (21 of 30; 95% CI 54 to 86%) even at a small optimization budget. A two-annotator audit confirms the attacks leave the harm intact: every one of 80 sampled flips still contained the harmful content. Because a large and growing share of reported ASR comes from LLM-judges, many such numbers are unreliable both on average and under deliberate pressure. We recommend that papers report judge precision and recall on a human-labeled slice, report ASR corrected for judge precision, and include an adversarial check of the judge. Our code is released.

Related papers

• Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts [2.583375572224943]
  We measure flip rates with paired significance tests and measured noise floors.<n>Findings are precise rather than universal: most judges barely move, but specific judges harbor cheaply exploitable blind spots.<n>We release the dataset, wrappers, code, and per-verdict labels.
  arXiv  Detail & Related papers  (2026-09-08T04:27:22Z)
• trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories [0.0]
  Outcome-only evaluation is the production default for LLM agents.<n>We measure that blind spot where ground truth is known by construction.<n>We argue that judge evaluations must stratify recall by outcome survival.
  arXiv  Detail & Related papers  (2026-08-29T10:14:05Z)
• Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG [0.0]
  We introduce Eval-Pair Matrix, a controlled meta evaluation protocol for source-grounded RAG.<n>We induce one hidden answer-causal contradiction per record, generate answers from perturbed passages with GPT, Grok, and Gemini models, and then use the same models as blind judges to evaluate each answer against the original passages.<n>The experiment contains 300 core records, 897 labeled generator outputs, and 2,683 judge verdicts in a crossed 3 x 3 matrix.
  arXiv  Detail & Related papers  (2026-07-12T07:39:45Z)
• More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges [4.167333498582309]
  Training a language model against its own reference-free judgments assumes a model's verdict on a shown answer tracks correctness.<n>We show it fails structurally: conditioned on a candidate, a judge scores plausibility, not correctness, leaving false-positive basins a policy learns to exploit.
  arXiv  Detail & Related papers  (2026-07-07T06:59:30Z)
• On the Adversarial Robustness of Multimodal LLM Judges [49.80009278197918]
  We introduce RobustMLLMJudge, the first framework for evaluating the adversarial robustness of general-purpose MLLMs when functioning as judges.<n>Using RobustMLLMJudge, we reveal that different MLLM judges are highly vulnerable to score-inflating adversarial attacks.<n>We propose MGSIA, a novel method that bypasses these constraints to enable more effective and transferable attacks on MLLM judges.
  arXiv  Detail & Related papers  (2026-06-14T05:30:20Z)
• Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines [0.22843885788439797]
  Continuous evaluation of products relies on a strong LLM judge treated as ground truth.<n>Judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores.<n>We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard.
  arXiv  Detail & Related papers  (2026-06-13T21:12:27Z)
• Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents [45.148328075418156]
  We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems it catches.<n>Our blind-spot taxonomy shows the failure is structured, not random.<n>For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.
  arXiv  Detail & Related papers  (2026-06-09T02:11:01Z)
• Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges [65.92623263645139]
  We introduce BITE, a black-box adversarial framework that learns semantics-preserving edits to mislead an LLM judge.<n>BITE achieves an attack success rate exceeding 65% and raises scores by 1-2 points on a 9-point scale.<n>Our findings expose a fundamental weakness in the LLM-as-a-judge paradigm and motivate robust, attack-aware evaluation.
  arXiv  Detail & Related papers  (2026-05-24T05:24:09Z)
• How Sensitive Are Safety Benchmarks to Judge Configuration Choices? [5.94231111588812]
  Safety benchmarks such as HarmBench rely on judges to classify model responses as harmful or safe.<n>We show that the combination of judge model and judge prompt is typically treated as a fixed implementation detail.<n>We find that prompt wording alone, holding the judge model fixed, shifts measured harmful-response rates by up to 24.2 percentage points.
  arXiv  Detail & Related papers  (2026-04-27T05:59:59Z)
• JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems [0.0]
  Large language models are increasingly deployed as automated judges for evaluating other models.<n>We introduce JudgeSense, a framework and benchmark for quantifying this property via the Judge Sensitivity Score (JSS)<n>On factuality, all judges cluster near JSS about 0.63, driven by a polarity-inverted prompt artifact.<n>We release code, decision logs, and a validated paraphrase dataset to support standardized JSS reporting.
  arXiv  Detail & Related papers  (2026-04-26T00:08:30Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• Evaluative Fingerprints: Stable and Systematic Differences in LLM Evaluator Behavior [0.0]
  We find that judges are consistent, but not with each other; they are consistent with themselves.<n>Across 3,240 evaluations, inter-judge agreement is near-zero.<n>Averaging judges' scores produces a synthetic verdict that corresponds to no judge's actual values.
  arXiv  Detail & Related papers  (2026-01-08T17:02:22Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges [48.053949045598344]
  evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively.<n>We propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks.<n>We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors.
  arXiv  Detail & Related papers  (2023-10-26T17:48:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.