SecureCodeRL: Security-Aware Reinforcement Learning for Code Generation with Partial-Credit Rewards

• URL: http://arxiv.org/abs/2601.01184v1
• Date: Sat, 03 Jan 2026 13:36:36 GMT
• Title: SecureCodeRL: Security-Aware Reinforcement Learning for Code Generation with Partial-Credit Rewards
• Abstract summary: This paper presents SecureCodeRL, a reinforcement learning pipeline for security-aware code generation.<n>The key idea is a partial-credit functional reward that assigns intermediate scores for validity, successful execution, and producing output.<n>Although Bandit findings were absent in this small evaluation, the security term is integrated into training to discourage insecure shortcuts when they appear.
• Score: 1.5377279217726239
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) can generate plausible code, but in settings that require exact stdin/stdout behavior they frequently produce programs that compile yet fail tests, and in some cases they introduce security-sensitive patterns. This paper presents SecureCodeRL, a reinforcement learning (RL) pipeline for security-aware code generation that optimizes a combined reward R = αRfunc + \b{eta}Rsec. The key idea is a partial-credit functional reward that assigns intermediate scores for syntactic validity, successful execution, and producing output, reducing reward sparsity that otherwise stalls learning on competitive programming style tasks. I evaluate supervised fine-tuning (SFT) and PPO variants on a small held-out prompt set from APPS+ and observe that PPO with partial credit (using a continued-training variant) improves syntax validity from 45% (SFT) to 60% and achieves the only non-zero test success signal in this pilot evaluation (5% at-least-one-test-pass), while remaining 100% clean under Bandit static analysis. Although Bandit findings were absent in this small evaluation, the security term is integrated into training to discourage insecure shortcuts when they appear.

Related papers

• Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation [18.470572796807094]
  Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers.<n>We propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement.<n>PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus.
  arXiv  Detail & Related papers  (2026-09-08T17:54:13Z)
• Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproduction [62.42945715963878]
  Repository-level vulnerability reproduction is a demanding software engineering (SE) task.<n>Recent LLM agents can often execute these steps when the approach is correct, yet they still fail by choosing the wrong strategy.<n>This paper argues that strategy, rather than the full action trajectory, is the right learning unit for such SE agents.
  arXiv  Detail & Related papers  (2026-07-02T06:27:27Z)
• When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study [77.64957118012423]
  Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in large language model reasoning.<n>Recent work addresses this by deriving rewards from a model's own signals, such as majority voting or confidence-based scores.<n>We present a systematic empirical study of intrinsic reward methods for code generation.
  arXiv  Detail & Related papers  (2026-06-18T19:15:50Z)
• VIMPO: Value-Implicit Policy Optimization for LLMs [106.88933849641272]
  Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token.<n>Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability.<n>We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from optimality conditions of KL-regularized reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-18T09:44:12Z)
• SecRL-Prune: Structured Reinforcement Learning-Based Pruning of CodeLLMs for Preserving Adversarial Code Mutation [0.4078247440919472]
  SecRL-Prune is a structured pruning framework for large code language models (CodeLLMs)<n>It learns a layer-wise pruning policy with reinforcement learning using a teacher-student KL-divergence reward.<n>We evaluate SecRL-Prune on HumanEval using pass@k for execution correctness and var@k for code diversity across three 7B CodeLLMs at 10-30% compression.
  arXiv  Detail & Related papers  (2026-06-04T14:55:14Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learning [19.054149750597933]
  MIST-RL (Mutation-based Incremental Suite Testing via Reinforcement Learning) is a framework that shifts the focus to "scaling-by-utility"<n>We introduce a novel incremental mutation reward combined with dynamic penalties, which incentivizes the model to discover new faults while it suppresses functionally equivalent assertions.<n>Experiments on HumanEval+ and MBPP+ demonstrate that MIST-RL outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T03:22:44Z)
• CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions [8.163435280190027]
  Existing benchmarks often lack coverage for subtle corner cases, allowing incorrect solutions to pass.<n>CodeHacker generates adversarial test cases that expose latent vulnerabilities in program submissions.<n>Experiments demonstrate that CodeHacker significantly improves the True Negative Rate (TNR) of existing datasets.
  arXiv  Detail & Related papers  (2026-02-23T05:59:30Z)
• Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models [59.6715047267181]
  Small reasoning models (SRMs) are prone to hallucinations, especially in intermediate reasoning steps.<n>Existing mitigation methods based on online reinforcement learning rely on outcome-based rewards or coarse-grained chain-of-thought evaluation.<n>We propose Faithfulness-Aware Step-Level Reinforcement Learning (FaithRL), introducing step-level supervision via explicit faithfulness rewards from a process reward model.
  arXiv  Detail & Related papers  (2026-02-05T17:15:12Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• Aletheia: What Makes RLVR For Code Verifiers Tick? [51.371034079170435]
  Verifiers trained via Reinforcement Learning from Verifiable Rewards (RLVR) are a prominent fixture of the Large Language Model (LLM) post-training pipeline.<n>Code verifiers remain valuable toward judging model outputs in scenarios where execution feedback is hard to obtain.<n>We examine components of the RLVR-based verifier training recipe widely credited for its success.
  arXiv  Detail & Related papers  (2026-01-17T22:30:45Z)
• VeRPO: Verifiable Dense Reward Policy Optimization for Code Generation [43.206705536310245]
  We introduce textbfVeRPO (textbfVerifiable Dtextbfense textbfReward textbfPolicy textbfOptimization), a novel RL framework for code generation that synthesizes textitrobust and dense rewards fully grounded in verifiable execution feedback.<n>VeRPO consistently outperforms outcome-driven and RM-based baselines, achieving up to +8.83% gain in pass@1 with negligible time cost ( 0.02%) and zero
  arXiv  Detail & Related papers  (2026-01-07T02:29:49Z)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding [54.72617309922891]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a core paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs)<n>Previous practice requires the LLM to sequentially generate solutions and self-verifications using two separate prompt templates, which significantly reduces efficiency.<n>We propose LaSeR (Reinforcement Learning with Last-Token Self-Rewarding), an algorithm that simply augments the original RLVR loss with a MSE loss.
  arXiv  Detail & Related papers  (2025-10-16T17:55:11Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models [0.0]
  Generative AI is gaining increasing attention in software engineering.<n>Unit tests constitute the majority of test cases and are often schematic.<n>This paper investigates the impact of code context and prompting strategies on the quality and adequacy of unit tests.
  arXiv  Detail & Related papers  (2025-07-18T11:23:17Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learning [14.852334980733369]
  Credit assignmen, disentangling each agent's contribution to a shared reward, is a critical challenge in cooperative multi-agent reinforcement learning.<n>We introduce Temporal-Agent Reward Redistribution (TAR$2$), an approach that decouples credit modeling from this constraint.<n>We demonstrate that this method is equivalent to a valid Potential-Based Reward Shaping (PBRS), which guarantees the optimal policy is preserved regardless of model accuracy.
  arXiv  Detail & Related papers  (2025-02-07T12:07:57Z)
• DeVAIC: A Tool for Security Assessment of AI-generated Code [5.383910843560784]
  DeVAIC (Detection of Vulnerabilities in AI-generated Code) is a tool to evaluate the security of AI-generated Python code.
  arXiv  Detail & Related papers  (2024-04-11T08:27:23Z)
• $\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis [39.742755916373284]
  Program synthesis aims to create accurate, executable programs from problem specifications. Recent studies have leveraged the power of reinforcement learning (RL) in conjunction with large language models (LLMs) Our work explores the feasibility of value-based approaches, leading to the development of our $mathcalB$-Coder.
  arXiv  Detail & Related papers  (2023-10-04T21:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.