Security Tests as Executable Specifications for LLM Code Generation: Benefits, Trade-offs, and Coverage Limits

• URL: http://arxiv.org/abs/2608.09740v2
• Date: Tue, 11 Aug 2026 02:30:40 GMT
• Title: Security Tests as Executable Specifications for LLM Code Generation: Benefits, Trade-offs, and Coverage Limits
• Abstract summary: We develop a controlled test-feedback scaffold that separates three factors: whether tests are shown upfront, whether failed executions trigger revision, and how failures are selected and represented.<n>Across 2,705 trajectories, 31 task instances, three secure-code benchmarks, 16 CWE categories, and two model families, showing all visible tests upfront increases hidden functional-and-security joint success by 19.3 percentage points on average.<n>In shared-candidate comparisons, structured feedback repairs 80 initially unsuccessful candidates with no joint regressions; fixed raw feedback repairs 83 but causes three regressions.
• Score: 14.44743150931186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) can generate functionally useful code that remains vulnerable, while security-focused interventions may break intended behavior. We investigate security tests as executable specifications both before generation and during iterative repair. We develop SecTDD, a controlled test-feedback scaffold that separates three factors: whether tests are shown upfront, whether failed executions trigger revision, and how failures are selected and represented. The evaluation uses behavior-partitioned visible and hidden tests and byte-identical initial candidates for repair comparisons. Across 2,705 trajectories, 31 task instances, three secure-code benchmarks, 16 CWE categories, and two model families, showing all visible tests upfront increases hidden functional-and-security joint success by 19.3 percentage points on average, but improves only seven of nine benchmark-model conditions and harms two. In shared-candidate comparisons, structured feedback repairs 80 initially unsuccessful candidates with no joint regressions; fixed raw feedback repairs 83 but causes three regressions. Structured and raw feedback are otherwise nearly indistinguishable head-to-head (six wins, six losses, and 453 ties). Candidates that pass all visible tests still fail hidden behavior families under every common regime. These results show that executable feedback can repair secure-code generation, but its benefits depend on the model, task, feedback entry point, and especially test coverage.

Related papers

• Guardrailed Meta-Agent Loops: Stress-Testing Policy Pinning, Budget Bounds, and Crash Recovery [7.414694666903067]
  GuardrailLoop is a simulation-based testbed that makes three operational contracts jointly testable.<n>A hash-pinned policy fixes goals, scope, evaluation identity, budget, and release conditions.<n>It separates useful adaptation, state recovery, and repeated execution.
  arXiv  Detail & Related papers  (2026-09-10T21:22:08Z)
• ExecCritic: Learn to Test, Test to Improve for Coding Agents [68.42713285082912]
  Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue.<n>We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it.<n>The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests.
  arXiv  Detail & Related papers  (2026-09-08T17:53:37Z)
• ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts [0.0]
  Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes.<n>We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw.
  arXiv  Detail & Related papers  (2026-08-23T17:09:02Z)
• TaPR: Test-Aware Policy Refinement for Feedback-Conditioned Code Generation [6.268849049716121]
  Test-aware Policy Refinement (TaPR) is a framework that transforms execution feedback into a dense per-turn test-pass-ratio reward.<n>Across six models on 219 code-generation problems from LiveCodeBench, TaPR improves the pooled three-turn success rate (Pass@3) by 2.44 percentage points.
  arXiv  Detail & Related papers  (2026-08-01T07:35:06Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• Regression Accumulation in Multi-Turn LLM Programming Conversations [25.94870617800846]
  regression accumulation appears across all six models.<n>Cross-Turn Conflict, where later code conflicts with earlier requirements, is the main failure class.<n> Verification Gate is the only strategy that consistently improves all models.
  arXiv  Detail & Related papers  (2026-07-02T08:15:40Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• An Iterative Test-and-Repair Framework for Competitive Code Generation [9.137158235106943]
  Large language models (LLMs) have made remarkable progress in code generation, but competitive programming remains a challenge.<n>Recent training-based methods have improved code generation by using reinforcement learning (RL) with execution feedback.<n>The more recent framework CURE further incorporates test generation into the training process, jointly training a Coder and a Tester within a single model.
  arXiv  Detail & Related papers  (2026-04-07T08:00:54Z)
• Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
  Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
  arXiv  Detail & Related papers  (2026-04-02T01:13:40Z)
• Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety [32.40159096400358]
  Large Language Model (LLM) agents increasingly act through external tools, making their safety contingent on tool-call rather than text generation alone.<n>While recent benchmarks evaluate agents across diverse environments and risk categories, a fundamental question remains unanswered: how complete are existing test suites, and what unsafe interaction patterns persist even after an agent passes the benchmark?<n>We propose SafeAudit, a meta-audit framework that addresses this gap through two contributions. First, an enumerator that systematically generates test cases by enumerating valid tool-call and diverse user scenarios. Second, we introduce rule-resistance, a non-semantic, quantitative metric
  arXiv  Detail & Related papers  (2026-03-18T20:06:47Z)
• A Framework for Creating Non-Regressive Test Cases via Branch Consistency Analysis Driven by Descriptions [9.141981611891715]
  DISTINCT is a Description-guided, branch-consistency analysis framework.<n>It transforms Large Language Model (LLM)-based generators into fault-aware test generators.<n>It achieves an average improvement of 14.64% in Compilation Success Rate (CSR) and 6.66% in Passing Rate (PR)
  arXiv  Detail & Related papers  (2025-06-09T07:05:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.