Auditing Reward Hackability in Code RL Training Environments

• URL: http://arxiv.org/abs/2606.16062v1
• Date: Sun, 14 Jun 2026 23:31:42 GMT
• Title: Auditing Reward Hackability in Code RL Training Environments
• Abstract summary: We measure the rate at which code RL environments accept incorrect solutions as correct.<n>On a 49-task sample of SWE-bench Verified, 28.5% of tasks have test suites weak enough that a Docker-verified incorrect patch passes them.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We measure the rate at which code RL environments accept incorrect solutions as correct. On a 49-task sample of SWE-bench Verified, 28.5% of tasks have test suites weak enough that a Docker-verified incorrect patch passes them. On 20 R2E-Gym tasks across 6 repositories, the same pipeline at single-shot exploit generation yields 25.0%. A random-effects meta-analysis over 134 frontier model submissions to SWE-bench Verified finds, within the same human-rated difficulty stratum, model Pass@1 is +14.14 percentage points higher on flagged-hackable tasks than on robust ones (95% CI [+11.80, +16.48]; one-sided p < 10^-6; I^2 = 0%; 123 of 134 models positive). We then describe a procedure for hardening the broken tasks. An inline LLM judge with a Docker gold-sanity gate runs each generated test against the gold solution before the judge is consulted. On the 11 broken tasks in the audit, the gate flags 65 of 105 decisive LLM-generated tests as failing on the gold patch itself, a 61.9% per-augmentation defect rate the LLM judge alone misses. With diversity-biased retry, the loop converges 9 of 11 tasks to a gated upgrade.

Related papers

• ExecCritic: Learn to Test, Test to Improve for Coding Agents [68.42713285082912]
  Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue.<n>We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it.<n>The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests.
  arXiv  Detail & Related papers  (2026-09-08T17:53:37Z)
• LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks [3.789882665982407]
  Multi-agent large language model (LLM) systems can improve reasoning by spending more computation.<n>We isolate this decision by running every problem under four protocols while holding the solver fixed within each setting.
  arXiv  Detail & Related papers  (2026-08-14T22:35:37Z)
• CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents [8.150687615894268]
  We introduce CodeGrep, a 14B retrieval agent trained end-to-end with GRPO to issue parallel, glob, and read tool calls and return candidate files to a frozen downstream coding agent.<n>On all 500 SWE-Bench Verified instances, CodeGrep preserves resolve rate while substantially improving efficiency: 27.0% versus 25.8% for the no-retrieval baseline, with 15% fewer rounds and 19% fewer tokens on resolved instances.
  arXiv  Detail & Related papers  (2026-08-06T11:07:42Z)
• Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug? [1.6787220460106658]
  BSG-VA captures each validation command at its exact working-tree state, extracts a test-only patch, and replays the command on the original buggy code.<n>The captured outcome and the replay results assign every event an evidence role, from gold-aligned bug-discriminating through regression-only to misleading.<n>A three-arm experiment tests whether returning the B-replay outcome to the agent changes this pattern.
  arXiv  Detail & Related papers  (2026-07-30T22:32:14Z)
• Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes [55.114939648705395]
  Large language models (LLMs) have made automated program repair increasingly practical for real-world bugs.<n>Bug reproduction tests (BRTs) help close this gap by turning a bug report into an executable, bug-specific signal.<n>We propose CoHarden, a co-generation framework that uses the Lax signal as an in-loop convergence criterion.
  arXiv  Detail & Related papers  (2026-07-22T07:30:07Z)
• Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent [0.0]
  We study one production system whose architecture installs verification loops.<n>We instrument the loop end-to-end, yielding an empirical verifier confusion matrix.
  arXiv  Detail & Related papers  (2026-07-19T03:20:33Z)
• Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents [99.67681154381803]
  Test-time Rule Acquisition and Compiled Enforcement (TRACE) is a skill-layer pipeline for coding-agent runtimes.<n>Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections.
  arXiv  Detail & Related papers  (2026-06-11T10:43:40Z)
• Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports [0.0]
  invalid bug reports do not require code changes, and are resolved with a no-code fix.<n>Our goal is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification.<n>We study how different configurations perform on a gold-standard benchmark.
  arXiv  Detail & Related papers  (2026-05-17T17:45:13Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents [2.1942030377331245]
  Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets.<n>We introduce MOSAIC-Bench, a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates.<n>We show that nine production coding agents compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs.
  arXiv  Detail & Related papers  (2026-05-05T16:38:23Z)
• Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
  Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
  arXiv  Detail & Related papers  (2026-04-02T01:13:40Z)
• AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling [12.42698406379544]
  AgentHER recovers lost training signal by adapting the Hindsight Experience principle to natural-language agent trajectories.<n>It converts discarded failures into high-quality SFT, DPO, and ShareGPT training data.<n>On WebArena Replay (Zhou et al., 2024) and ToolBench (Qin et al., 2024), AgentHER improves over success-only SFT by +7.1-11.7 pp across four model families.
  arXiv  Detail & Related papers  (2026-03-22T18:36:58Z)
• Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation [0.0]
  This study analyzes 319 Large Language Models (LLMs)generated security patches across 64 Java vulnerabilities from the Vul4J benchmark.<n>Using tri-axis evaluation, the analysis reveals that only 24.8% of patches achieve full correctness, while 51.4% fail both security and functionality.<n>The proposed Security Repair Score (SRS) quantifies this gap, showing LLMs preserve functionality (mean 0.832) but struggle with security (mean 0.251)
  arXiv  Detail & Related papers  (2026-03-10T05:34:56Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents [52.20768003832476]
  We analyze execution traces on $$-Bench (Airline/Retail) and SWE-Bench Verified.<n>We formalize emphdecisive deviations, earliest action, level divergences that flip success to failure.<n>We introduce cm, a model-agnostic, gradient-free, test-time safeguard.
  arXiv  Detail & Related papers  (2025-11-26T01:28:22Z)
• Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis [57.40527331817245]
  Test oracle generation in non-regression testing is a longstanding challenge in software engineering.<n>We introduce Nexus, a novel multi-agent framework to address this challenge.
  arXiv  Detail & Related papers  (2025-10-30T12:20:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.