Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code

• URL: http://arxiv.org/abs/2606.11817v1
• Date: Wed, 10 Jun 2026 08:50:59 GMT
• Title: Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code
• Abstract summary: Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code.<n>We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code.<n>We propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints.
• Score: 6.9750665065938975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications.

Related papers

• The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs [12.213106649470973]
  Large Language Models (LLMs) are increasingly used for source code generation despite their outputs often exhibiting security vulnerabilities.<n>We show that specific syntactic elements, such as constraints, guards, and concept bindings, and their position within the prompt consistently affect the likelihood of generating insecure code.<n>These findings identify prompt syntax as a concrete security control surface and provide actionable guidance for reducing vulnerability risk in LLM-assisted development.
  arXiv  Detail & Related papers  (2026-07-17T13:20:21Z)
• Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Prompts [5.718926328180089]
  This paper introduces a jailbreaking approach, CodeJailbreaker, designed to uncover safety concerns in code generation.<n> Experiments on the recently-released RMCBench benchmark demonstrate that CodeJailbreaker markedly surpasses the conventional jailbreaking strategy.
  arXiv  Detail & Related papers  (2025-03-23T06:06:12Z)
• From Solitary Directives to Interactive Encouragement! LLM Secure Code Generation by Natural Language Prompting [24.27542373791212]
  This work introduces SecCode, a framework that leverages an innovative interactive encouragement prompting (EP) technique for secure code generation with textitonly NL prompts. SecCode functions through three stages: 1) Code Generation using NL Prompts; 2) Code Vulnerability Detection and Fixing, utilising our proposed encouragement prompting; 3) Vulnerability Cross-Checking and Code Security Refinement.
  arXiv  Detail & Related papers  (2024-10-18T09:32:08Z)
• HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Data [60.75578581719921]
  Large language models (LLMs) have shown great potential for automatic code generation. Recent studies highlight that many LLM-generated code contains serious security vulnerabilities. We introduce HexaCoder, a novel approach to enhance the ability of LLMs to generate secure codes.
  arXiv  Detail & Related papers  (2024-09-10T12:01:43Z)
• ShadowCode: Towards (Automatic) External Prompt Injection Attack against Code LLMs [56.46702494338318]
  This paper introduces a new attack paradigm: (automatic) external prompt injection against code-oriented large language models.<n>We propose ShadowCode, a simple yet effective method that automatically generates induced perturbations based on code simulation.<n>We evaluate our method across 13 distinct malicious objectives, generating 31 threat cases spanning three popular programming languages.
  arXiv  Detail & Related papers  (2024-07-12T10:59:32Z)
• Constrained Decoding for Secure Code Generation [9.007821185927277]
  This paper introduces a new benchmark, CodeGuard+, to measure Code LLMs' ability to generate both secure and correct code. We show that the state-of-the-art defense technique, prefix tuning, may not be as strong as previously believed, since it generates secure code but sacrifices functional correctness. We propose new constrained decoding techniques to generate secure code.
  arXiv  Detail & Related papers  (2024-04-30T21:52:19Z)
• CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion [117.178835165855]
  This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs. Our studies reveal a new and universal safety vulnerability of these models against code input. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization.
  arXiv  Detail & Related papers  (2024-03-12T17:55:38Z)
• CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models [49.60006012946767]
  We propose CodeChameleon, a novel jailbreak framework based on personalized encryption tactics. We conduct extensive experiments on 7 Large Language Models, achieving state-of-the-art average Attack Success Rate (ASR) Remarkably, our method achieves an 86.6% ASR on GPT-4-1106.
  arXiv  Detail & Related papers  (2024-02-26T16:35:59Z)
• Weak-to-Strong Jailbreaking on Large Language Models [92.52448762164926]
  Large language models (LLMs) are vulnerable to jailbreak attacks.<n>Existing jailbreaking methods are computationally costly.<n>We propose the weak-to-strong jailbreaking attack.
  arXiv  Detail & Related papers  (2024-01-30T18:48:37Z)
• DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructions [27.489622263456983]
  We introduce DeceptPrompt, an algorithm that can generate adversarial natural language instructions that drive the Code LLMs to generate functionality correct code with vulnerabilities. When applying the optimized prefix/suffix, the attack success rate (ASR) will improve by average 50% compared with no prefix/suffix applying.
  arXiv  Detail & Related papers  (2023-12-07T22:19:06Z)
• A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily [51.63085197162279]
  Large Language Models (LLMs) are designed to provide useful and safe responses. adversarial prompts known as 'jailbreaks' can circumvent safeguards. We propose ReNeLLM, an automatic framework that leverages LLMs themselves to generate effective jailbreak prompts.
  arXiv  Detail & Related papers  (2023-11-14T16:02:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.