Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests

• URL: http://arxiv.org/abs/2605.28734v1
• Date: Wed, 27 May 2026 16:55:15 GMT
• Title: Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests
• Abstract summary: Coding-specialized models should clear a higher refusal bar than general-purpose chat models.<n>Eight corpora (ASTRA, CySecBench, AdvBench/harmful_behaviors, JailbreakBench, MalwareBench, RedCode, RMCBench, Scam2Prompt) are consolidated and classified under a five-judge consensus protocol.<n>The bank is the validated instrument the field has lacked: a reliability-quantified basis for testing whether coding models meet the stricter refusal standard.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A general-purpose language model that answers a harmful question returns text; a coding model that complies with a malicious request can return a working weapon -- a keylogger, a ransomware stub, an exploit that runs as written. This asymmetry in the severity of a single act of compliance implies coding-specialized models should clear a higher refusal bar than general-purpose chat models, not a lower one, yet the field cannot presently tell whether they do. Refusal benchmarks for malicious code are fragmented: they mix requests for executable software (ready-to-run weapons) with requests for harmful security knowledge (information a human must still operationalise) and report refusal rates over non-comparable corpora, so no single statistic measures the property that actually matters. This paper introduces an expanded consensus-labeled prompt bank that distinguishes between these two request types and provides a construct-stable substrate for cross-corpus coding-model compliance measurement. Eight corpora (ASTRA, CySecBench, AdvBench/harmful_behaviors, JailbreakBench, MalwareBench, RedCode, RMCBench, Scam2Prompt) are consolidated and classified under a five-judge consensus protocol (6,675 prompts x 5 judges = 33,375 calls). The panel reaches Fleiss' kappa = 0.767 [95% CI 0.755, 0.777] ("substantial"); 95.0% of prompts draw at least four agreeing judges, 76.9% are unanimous, and the panel reproduces the earlier four-corpus release at Cohen's kappa = 0.952 on the 3,133 shared prompts. The released bank comprises 4,748 consensus-CODE prompts (executable malicious code requests) and 1,923 consensus-KNOWLEDGE prompts (harmful security knowledge requests). The bank is the validated instrument the field has lacked: a reliability-quantified basis for testing whether coding models meet the stricter refusal standard their executable output demands.

Related papers

• Same Request, Different Boundary: Evaluating Cybersecurity Assistance across Conversational Contexts [25.330089031016076]
  Large Language Models (LLMs) can solve complex problems, but their misuse in high-risk domains can lead to severe consequences.<n>We introduce 3R-Bench, a benchmark of 150 real-world cybersecurity requests augmented with two adversarial conversational settings.
  arXiv  Detail & Related papers  (2026-09-01T02:16:56Z)
• Stealing Reasoning Traces from Proprietary LLM APIs [84.58379404903394]
  We develop a scalable decryption jailbreak for encrypted reasoning traces.<n>It circumvents anti-distillation mechanisms, allowing adversaries to extract a proprietary model's reasoning.<n>It inadvertently reveals hazardous information hidden within the reasoning process.
  arXiv  Detail & Related papers  (2026-08-10T17:24:50Z)
• Cross-Cutting Security Analysis of LLM-Generated Code via Metamorphic Testing and Association Rule Mining [0.7844254062143549]
  Large language models (LLMs) frequently generate code with security vulnerabilities, yet these weaknesses are rarely isolated.<n>We present a framework that combines security-oriented Metamorphic Relations (MRs) with Association Rule (AR) mining to detect vulnerabilities in LLM-generated code.
  arXiv  Detail & Related papers  (2026-07-13T19:10:10Z)
• SafeClawBench: Separating Semantic, Audit-Evidence, and Sandbox Harm in Tool-Using LLM Agents [31.978790013349947]
  SafeClawBench is a benchmark for tool-using agent security with 600 controlled adversarial tasks.<n>We evaluate five agent endpoints under four prompt-level policies.<n>We find that these endpoints capture different failure modes.
  arXiv  Detail & Related papers  (2026-06-16T18:04:45Z)
• RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Prompts [0.0]
  Strict refusal rates span 0.1% to 94.6% on identical prompts.<n>Nine of 18 frontier models exhibit a hedge-but-help partial-compliance pattern at dual-use tier that binary refusal metrics cannot detect.
  arXiv  Detail & Related papers  (2026-05-20T09:53:31Z)
• Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025) [0.0]
  Existing surveys treat code security, jailbreak taxonomy, or vulnerability detection as the central object and mention these corpora only in passing.<n>This paper treats the prompt datasets themselves as the unit of analysis.<n>The synthesis surfaces three recurring methodological gaps.
  arXiv  Detail & Related papers  (2026-05-19T18:05:51Z)
• Hallucination as Exploit: Evidence-Carrying Multimodal Agents [10.441697487723568]
  Multimodal agents increasingly choose tool calls from screenshots, documents, and webpages, where a false perceptual claim can turn from an answer-quality error into an authorization failure.<n>We propose evidence-carrying multimodal agents (ECA), which treat free-form model text as inadmissible evidence.<n>ECA converts opaque model belief into auditable residuals at the verifier, schema, and implementation levels.
  arXiv  Detail & Related papers  (2026-05-18T23:40:43Z)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents [2.1942030377331245]
  Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets.<n>We introduce MOSAIC-Bench, a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates.<n>We show that nine production coding agents compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs.
  arXiv  Detail & Related papers  (2026-05-05T16:38:23Z)
• A Validated Prompt Bank for Malicious Code Generation: Separating Executable Weapons from Security Knowledge in 1,554 Consensus-Labeled Prompts [0.0]
  This paper introduces a weapons-versus-knowledge classification axis, operationalized through a five-model consensus protocol.<n>It applies it to 3,133 prompts drawn from four public benchmarks, yielding a 1,554-prompt consensus-CODE bank.<n>All 3,133 prompts reached the 3-of-5 threshold, so the consensus pipeline produced zero ambiguity-excluded prompts.
  arXiv  Detail & Related papers  (2026-05-04T21:42:10Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• Decompiling Smart Contracts with a Large Language Model [51.49197239479266]
  Despite Etherscan's 78,047,845 smart contracts deployed on (as of May 26, 2025), a mere 767,520 ( 1%) are open source.<n>This opacity necessitates the automated semantic analysis of on-chain smart contract bytecode.<n>We introduce a pioneering decompilation pipeline that transforms bytecode into human-readable and semantically faithful Solidity code.
  arXiv  Detail & Related papers  (2025-06-24T13:42:59Z)
• OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents [60.78202583483591]
  We introduce OS-Harm, a new benchmark for measuring safety of computer use agents.<n> OS-Harm is built on top of the OSWorld environment and aims to test models across three categories of harm: deliberate user misuse, prompt injection attacks, and model misbehavior.<n>We evaluate computer use agents based on a range of frontier models and provide insights into their safety.
  arXiv  Detail & Related papers  (2025-06-17T17:59:31Z)
• Certifying LLM Safety against Adversarial Prompting [70.96868018621167]
  Large language models (LLMs) are vulnerable to adversarial attacks that add malicious tokens to an input prompt.<n>We introduce erase-and-check, the first framework for defending against adversarial prompts with certifiable safety guarantees.
  arXiv  Detail & Related papers  (2023-09-06T04:37:20Z)
• Robust Encodings: A Framework for Combating Adversarial Typos [85.70270979772388]
  NLP systems are easily fooled by small perturbations of inputs. Existing procedures to defend against such perturbations provide guaranteed robustness to worst-case attacks. We introduce robust encodings (RobEn) that confer guaranteed robustness without making compromises on model architecture.
  arXiv  Detail & Related papers  (2020-05-04T01:28:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.