Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs

• URL: http://arxiv.org/abs/2605.29737v1
• Date: Thu, 28 May 2026 10:30:28 GMT
• Title: Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs
• Abstract summary: We show that mutations as small as a single-character change can flip generated code from secure to vulnerable.<n>Results show that input-handling flaws can be caught before generation while secure-defaults flaws require intervention during decoding.
• Score: 46.747768845221735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based coding assistants are seeing rapid adoption, offering substantial gains in developer productivity. As organizations increasingly ship code these agents produce, the security of that code becomes critical. Prior work has shown that minor prompt perturbations degrade the functional correctness of LLM-generated code, but whether they also compromise code security has remained unstudied. We apply token-level mutations to prompts across three models and five programming languages, and show that mutations as small as a single-character change can flip generated code from secure to vulnerable. Probing the models' hidden states reveals that this fragility is partially encoded in prompt representations, but unevenly so. Input-handling vulnerabilities, where the model omits validation or sanitization, are more predictable (mean AUC 0.753) than secure-defaults vulnerabilities, where insecure code stems from one local choice such as a weak algorithm or unsafe parameter (mean AUC 0.674). These results show that the threat model for LLM-assisted coding extends beyond prompt injection to ordinary prompt variation, and indicate that input-handling flaws can be caught before generation while secure-defaults flaws require intervention during decoding.

Related papers

• An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods [11.454777386934632]
  Large Language Models (LLMs) for automated code generation has enhanced software development efficiency, but often at the cost of security.<n>We present a comprehensive empirical evaluation of the security quality of LLM-generated code across five LLMs and four programming languages.<n>We introduce a weaknesses-aware zero-shot chain-of-thought (WA-0CoT) prompting strategy that enriches prompts with security context using CWE mappings to guide model reasoning.
  arXiv  Detail & Related papers  (2026-05-22T23:52:33Z)
• SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization [61.91729298584227]
  We introduce SecureForge, an automated pipeline that audits security risks of frontier models and produces auditing-informed secure system prompts.<n>SecureForge first identifies benign prompts that produce statically detectable vulnerabilities, and then amplifies them into a large synthetic prompt corpus of diverse scenarios.<n>On frontier models, SecureForge yields a statistically significant improvement in both unit test success and output security, with output vulnerabilities reduced by up to 48%.
  arXiv  Detail & Related papers  (2026-05-08T18:40:47Z)
• SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization [50.71047638695205]
  Reasoning language models (RLMs) are increasingly used in programming.<n>Yet, even state-of-the-art RLMs frequently introduce critical security vulnerabilities in generated code.<n>We present SecPI, a fine-tuning pipeline that teaches RLMs to internalize structured security reasoning.
  arXiv  Detail & Related papers  (2026-04-04T04:29:11Z)
• From Model to Breach: Towards Actionable LLM-Generated Vulnerabilities Reporting [43.57360781012506]
  We show that even the latest open-weight models are vulnerable in the earliest reported vulnerability scenarios.<n>We introduce a new severity metric that reflects the risk posed by an LLM-generated vulnerability.<n>To encourage the mitigation of the most serious and prevalent vulnerabilities, we use PE to define the Model Exposure (ME) score.
  arXiv  Detail & Related papers  (2025-11-06T16:52:27Z)
• TypePilot: Leveraging the Scala Type System for Secure LLM-generated Code [46.747768845221735]
  Large language Models (LLMs) have shown remarkable proficiency in code generation tasks across various programming languages.<n>Their outputs often contain subtle but critical vulnerabilities, posing significant risks when deployed in security-sensitive or mission-critical systems.<n>This paper introduces TypePilot, an agentic AI framework designed to enhance the security and robustness of LLM-generated code.
  arXiv  Detail & Related papers  (2025-10-13T08:44:01Z)
• A Mixture of Linear Corrections Generates Secure Code [20.94236753015922]
  Large language models (LLMs) have become proficient at sophisticated code-generation tasks, yet remain ineffective at reliably detecting or avoiding code vulnerabilities.<n>We find that current LLMs encode precise internal representations that distinguish vulnerable from secure code.<n>We develop an inference-time steering technique that subtly modulates the model's token-generation probabilities through a mixture of corrections.
  arXiv  Detail & Related papers  (2025-07-13T06:27:33Z)
• Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation [16.29310628754089]
  Large Language Models (LLMs) have become powerful tools for automated code generation.<n>LLMs often overlook critical security practices, which can result in the generation of insecure code.<n>This paper examines their inherent tendencies to produce insecure code, their capability to generate secure code when guided by self-generated vulnerability hints, and their effectiveness in repairing vulnerabilities when provided with different levels of feedback.
  arXiv  Detail & Related papers  (2025-06-28T23:24:33Z)
• HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Data [60.75578581719921]
  Large language models (LLMs) have shown great potential for automatic code generation. Recent studies highlight that many LLM-generated code contains serious security vulnerabilities. We introduce HexaCoder, a novel approach to enhance the ability of LLMs to generate secure codes.
  arXiv  Detail & Related papers  (2024-09-10T12:01:43Z)
• CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion [117.178835165855]
  This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs. Our studies reveal a new and universal safety vulnerability of these models against code input. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization.
  arXiv  Detail & Related papers  (2024-03-12T17:55:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.