Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety

• URL: http://arxiv.org/abs/2604.18487v1
• Date: Mon, 20 Apr 2026 16:37:27 GMT
• Title: Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety
• Abstract summary: Adversarial Humanities Benchmark evaluates whether model safety refusals survive a shift away from familiar harmful prompt forms.<n>The benchmark rewrites the same objectives through humanities-style transformations while preserving intent.<n>Results show that the original attacks record 3.84% attack success rate (ASR), while transformed methods range from 36.8% to 65.0%, yielding 55.75% overall ASR across 31 frontier models.
• Score: 1.638800175484975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Adversarial Humanities Benchmark (AHB) evaluates whether model safety refusals survive a shift away from familiar harmful prompt forms. Starting from harmful tasks drawn from MLCommons AILuminate, the benchmark rewrites the same objectives through humanities-style transformations while preserving intent. This extends literature on Adversarial Poetry and Adversarial Tales from single jailbreak operators to a broader benchmark family of stylistic obfuscation and goal concealment. In the benchmark results reported here, the original attacks record 3.84% attack success rate (ASR), while transformed methods range from 36.8% to 65.0%, yielding 55.75% overall ASR across 31 frontier models. Under a European Union AI Act Code-of-Practice-inspired systemic-risk lens, Chemical, biological, radiological and nuclear (CBRN) is the highest bucket. Taken together, this lack of stylistic robustness suggests that current safety techniques suffer from weak generalization: deep understanding of 'non-maleficence' remains a central unresolved problem in frontier model safety.

Related papers

• Safety Targeted Embedding Exploit via Refinement [1.384477926572109]
  We show that safety mechanisms aligned primarily on English cannot be assumed to generalize across multilingual inputs.<n>We introduce STEER, a gradient-guided attack that translates words into low-resource languages to suppress refusal while preserving harmful intent.<n>We argue that improving multilingual safety requires broader coverage during alignment and mechanisms that explicitly detect and abstain on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2026-07-02T08:17:57Z)
• Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety [2.661610409070365]
  We introduce Boiling the Frog, a benchmark that evaluates whether tool-using AI models are susceptible to incremental attacks.<n> Scenarios are organized through a three-level operational risk taxonomy.<n>Across a nine-model panel, aggregate strict attack success rate (ASR) is 44.4%.
  arXiv  Detail & Related papers  (2026-05-21T15:50:18Z)
• Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming [64.48633529149579]
  We propose a novel framework to expose the vulnerabilities of Vision-Language-Action (VLA) models against linguistic variations.<n>Our method reduces the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents.
  arXiv  Detail & Related papers  (2026-04-07T08:43:36Z)
• Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model [50.29667251847595]
  We show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models.<n>We propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space.<n>In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks.
  arXiv  Detail & Related papers  (2026-04-01T02:42:41Z)
• OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences [64.01706941950489]
  Current safety paradigms primarily target malicious intent or situational violations.<n>We propose shifting the safety frontier toward consequence-driven safety, a paradigm essential for the robust deployment of autonomous and embodied agents.<n>We develop the Consequence-Aware Safety Policy Optimization (CASPO) framework, which integrates the model's intrinsic reasoning as a dynamic reference for token-level self-distillation rewards.
  arXiv  Detail & Related papers  (2026-03-10T14:16:43Z)
• ProGuard: Towards Proactive Multimodal Safeguard [48.89789547707647]
  ProGuard is a vision-language proactive guard that identifies and describes out-of-distribution (OOD) safety risks.<n>We first construct a modality-balanced dataset of 87K samples, each annotated with both binary safety labels and risk categories.<n>We then train our vision-language base model purely through reinforcement learning to achieve efficient and concise reasoning.
  arXiv  Detail & Related papers  (2025-12-29T16:13:23Z)
• Replicating TEMPEST at Scale: Multi-Turn Adversarial Attacks Against Trillion-Parameter Frontier Models [0.0]
  This study employed the TEMPEST multi-turn attack framework to evaluate ten frontier models from eight vendors across 1,000 harmful behaviors.<n>Six models achieved 96% to 100% attack success rate (ASR), while four showed meaningful resistance, with ASR ranging from 42% to 78%.
  arXiv  Detail & Related papers  (2025-12-08T00:30:40Z)
• Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks [0.0]
  Large Language Model (LLM) safety guardrail models have emerged as a primary defense mechanism against harmful content generation.<n>This study evaluated ten publicly available guardrail models from Meta, Google, IBM, NVIDIA, Alibaba, and Allen AI across 1,445 test prompts spanning 21 attack categories.
  arXiv  Detail & Related papers  (2025-11-27T03:01:09Z)
• Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models [1.5401871453629499]
  We present evidence that adversarial poetry functions as a universal single-turn jailbreak technique for Large Language Models (LLMs)<n>Across 25 proprietary and open-weight models, curated poetic prompts yielded high attack-success rates (ASR), with some providers exceeding 90%.
  arXiv  Detail & Related papers  (2025-11-19T10:14:08Z)
• Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation [13.971909819796762]
  Large Language Models (LLMs) have achieved remarkable success across domains such as healthcare, education, and cybersecurity.<n>Embedding space poisoning is a subtle attack vector where adversaries manipulate the internal semantic representations of input data to bypass safety alignment mechanisms.<n>We propose ETTA, a novel framework that identifies and attenuates toxicity-sensitive dimensions in embedding space via linear transformations.
  arXiv  Detail & Related papers  (2025-07-08T03:01:00Z)
• Safety Pretraining: Toward the Next Generation of Safe AI [68.99129474671282]
  We present a data-centric pretraining framework that builds safety into the model from the start.<n>Our framework consists of four key steps: Safety Filtering, Safety Rephrasing, Native Refusal and Harmfulness-Tag annotated pretraining.<n>Our safety-pretrained models reduce attack success rates from 38.8% to 8.4% on standard LLM safety benchmarks with no performance on general degradation tasks.
  arXiv  Detail & Related papers  (2025-04-23T17:58:08Z)
• Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions [50.40122190627256]
  We introduce POATE, a novel jailbreak technique that harnesses contrastive reasoning to provoke unethical responses.<n>PoATE crafts semantically opposing intents and integrates them with adversarial templates, steering models toward harmful outputs with remarkable subtlety.<n>To counter this, we propose Intent-Aware CoT and Reverse Thinking CoT, which decompose queries to detect malicious intent and reason in reverse to evaluate and reject harmful responses.
  arXiv  Detail & Related papers  (2025-01-03T15:40:03Z)
• Improved Generation of Adversarial Examples Against Safety-aligned LLMs [72.38072942860309]
  Adversarial prompts generated using gradient-based methods exhibit outstanding performance in performing automatic jailbreak attacks against safety-aligned LLMs. In this paper, we explore a new perspective on this problem, suggesting that it can be alleviated by leveraging innovations inspired in transfer-based attacks. We show that 87% of the query-specific adversarial suffixes generated by the developed combination can induce Llama-2-7B-Chat to produce the output that exactly matches the target string on AdvBench.
  arXiv  Detail & Related papers  (2024-05-28T06:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.