Prompt Structure Redistributes, Not Reduces: An Empirical Analysis of Security-Weaknesses in LLM-Generated Python Code

• URL: http://arxiv.org/abs/2608.24857v1
• Date: Tue, 25 Aug 2026 17:43:47 GMT
• Title: Prompt Structure Redistributes, Not Reduces: An Empirical Analysis of Security-Weaknesses in LLM-Generated Python Code
• Abstract summary: We generate solutions with GPT-4o and LLaMA 3.1-8B under five prompt variants.<n>We evaluate them with Bandit and CodeQL along two axes: generation compliance and security weakness prevalence, severity, and CWE.
• Score: 3.919854840656461
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) increasingly generate code from natural-language prompts, making prompt engineering a key mechanism for shaping the security of generated software. Structured and security-oriented prompts are widely used to encourage safer code, yet their effects extend beyond whether detected weaknesses are simply present or absent. Using 424 security-sensitive Python tasks, we generate solutions with GPT-4o and LLaMA 3.1-8B under five prompt variants that progressively add structural and security guidance, and evaluate them with Bandit and CodeQL along two axes: generation compliance and security weakness prevalence, severity, and CWE distributions. Structured prompting substantially reduces refusals (e.g., GPT-4o invalid outputs drop from 338 of 424 to 37-52), enabling large-scale analysis, but security-oriented refinements do not consistently reduce overall weakness prevalence. For GPT-4o, stronger prompts primarily redistribute risk: high-severity findings fall (20.8% to 13.6%) while low-severity findings rise (32% to 43.5%); LLaMA shows weaker, less consistent shifts. We also observe security-driven semantic drift, where stricter prompts silently remove or rewrite explicitly requested unsafe constructs. Overall, prompt structure improves compliance but is an unreliable substitute for robust security controls in LLM-assisted development.

Related papers

• Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale [54.83971397981572]
  Language-model agents increasingly operate over complete software repositories.<n> cybersecurity evaluations measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code.<n>We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness.
  arXiv  Detail & Related papers  (2026-09-14T17:44:51Z)
• Cross-Cutting Security Analysis of LLM-Generated Code via Metamorphic Testing and Association Rule Mining [0.7844254062143549]
  Large language models (LLMs) frequently generate code with security vulnerabilities, yet these weaknesses are rarely isolated.<n>We present a framework that combines security-oriented Metamorphic Relations (MRs) with Association Rule (AR) mining to detect vulnerabilities in LLM-generated code.
  arXiv  Detail & Related papers  (2026-07-13T19:10:10Z)
• Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs [46.747768845221735]
  We show that mutations as small as a single-character change can flip generated code from secure to vulnerable.<n>Results show that input-handling flaws can be caught before generation while secure-defaults flaws require intervention during decoding.
  arXiv  Detail & Related papers  (2026-05-28T10:30:28Z)
• SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization [61.91729298584227]
  We introduce SecureForge, an automated pipeline that audits security risks of frontier models and produces auditing-informed secure system prompts.<n>SecureForge first identifies benign prompts that produce statically detectable vulnerabilities, and then amplifies them into a large synthetic prompt corpus of diverse scenarios.<n>On frontier models, SecureForge yields a statistically significant improvement in both unit test success and output security, with output vulnerabilities reduced by up to 48%.
  arXiv  Detail & Related papers  (2026-05-08T18:40:47Z)
• SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization [50.71047638695205]
  Reasoning language models (RLMs) are increasingly used in programming.<n>Yet, even state-of-the-art RLMs frequently introduce critical security vulnerabilities in generated code.<n>We present SecPI, a fine-tuning pipeline that teaches RLMs to internalize structured security reasoning.
  arXiv  Detail & Related papers  (2026-04-04T04:29:11Z)
• SCAFFOLD-CEGIS: Preventing Latent Security Degradation in LLM-Driven Iterative Code Refinement [12.69450437027072]
  This paper reveals the iterative refinement paradox: specification drift during multi-objective optimization causes security to degrade gradually over successive iterations.<n>We propose the SCAFFOLD-CEGIS framework that transforms security constraints from implicit prompts into explicit verifiable constraints.
  arXiv  Detail & Related papers  (2026-03-09T15:54:18Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security [63.41350337821108]
  We propose Secure Tug-of-War (SecTOW) to enhance the security of multimodal large language models (MLLMs)<n>SecTOW consists of two modules: a defender and an auxiliary attacker, both trained iteratively using reinforcement learning (GRPO)<n>We show that SecTOW significantly improves security while preserving general performance.
  arXiv  Detail & Related papers  (2025-07-29T17:39:48Z)
• Security Degradation in Iterative AI Code Generation -- A Systematic Analysis of the Paradox [0.11470070927586017]
  This paper analyzes security degradation in AI-generated code through a controlled experiment with 400 code samples.<n>We show a 37.6% increase in critical vulnerabilities after just five iterations.<n>We propose practical guidelines for developers to mitigate these risks.
  arXiv  Detail & Related papers  (2025-05-19T22:55:51Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.