Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards

• URL: http://arxiv.org/abs/2506.11425v2
• Date: Fri, 20 Jun 2025 23:32:06 GMT
• Title: Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards
• Abstract summary: We introduce Agent-RLVR, a framework that makes RLVR effective in challenging agentic settings.<n>Inspired by human pedagogy, Agent-RLVR introduces agent guidance, a mechanism that actively steers the agent towards successful trajectories.<n>We find that our guidance-augmented RLVR data is additionally useful for test-time reward model training, shown by further boosting pass@1 to 27.8%.
• Score: 8.390219473444446
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) has been widely adopted as the de facto method for enhancing the reasoning capabilities of large language models and has demonstrated notable success in verifiable domains like math and competitive programming tasks. However, the efficacy of RLVR diminishes significantly when applied to agentic environments. These settings, characterized by multi-step, complex problem solving, lead to high failure rates even for frontier LLMs, as the reward landscape is too sparse for effective model training via conventional RLVR. In this work, we introduce Agent-RLVR, a framework that makes RLVR effective in challenging agentic settings, with an initial focus on software engineering tasks. Inspired by human pedagogy, Agent-RLVR introduces agent guidance, a mechanism that actively steers the agent towards successful trajectories by leveraging diverse informational cues. These cues, ranging from high-level strategic plans to dynamic feedback on the agent's errors and environmental interactions, emulate a teacher's guidance, enabling the agent to navigate difficult solution spaces and promotes active self-improvement via additional environment exploration. In the Agent-RLVR training loop, agents first attempt to solve tasks to produce initial trajectories, which are then validated by unit tests and supplemented with agent guidance. Agents then reattempt with guidance, and the agent policy is updated with RLVR based on the rewards of these guided trajectories. Agent-RLVR elevates the pass@1 performance of Qwen-2.5-72B-Instruct from 9.4% to 22.4% on SWE-Bench Verified. We find that our guidance-augmented RLVR data is additionally useful for test-time reward model training, shown by further boosting pass@1 to 27.8%. Agent-RLVR lays the groundwork for training agents with RLVR in complex, real-world environments where conventional RL methods struggle.

Related papers

• EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learning [78.62829041672663]
  Reinforcement learning (RL) has emerged as a powerful paradigm for training Large Language Models (LLMs) as agents.<n>We propose EnvRL, a framework that incorporates environment dynamics learning into agentic RL via two auxiliary objectives.
  arXiv  Detail & Related papers  (2026-06-16T08:48:09Z)
• On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR [51.935533482549545]
  We show that RLVR may exhibit implicit reward overfitting to the training dataset.<n>The model can achieve satisfactory performance on the test set even when its rewards remain relatively low during the training process.
  arXiv  Detail & Related papers  (2026-05-07T16:30:28Z)
• Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training? [23.212983812736343]
  Agent2 RL-Bench is a benchmark for evaluating agentic RL post-training.<n>We find that supervised pipelines dominate agent-driven post-training under fixed budgets.
  arXiv  Detail & Related papers  (2026-04-12T09:35:27Z)
• RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback [54.39884046754265]
  RetroAgent is an online RL framework that enables agents to master complex interactive environments.<n>Our experiments show that RetroAgent achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-03-09T16:23:33Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement [21.073482007189504]
  Large language models (LLMs) have exhibited remarkable performance on complex reasoning tasks.<n> reinforcement learning under verifiable rewards (RLVR) is emerging as a principled framework for aligning model behavior with reasoning chains.<n>Despite its promise, RLVR remains prohibitively resource-intensive, requiring extensive reward signals and incurring substantial rollout costs during training.
  arXiv  Detail & Related papers  (2026-01-31T16:51:50Z)
• Demystifying Reinforcement Learning in Agentic Reasoning [90.3737088727791]
  We conduct a comprehensive and systematic investigation to demystify reinforcement learning in agentic reasoning.<n>We highlight our key insights: (i) replacing stitched synthetic trajectories with real end-to-end tool-use trajectories yields a far stronger SFT.<n> Exploration-friendly techniques are crucial for agentic RL, such as clip higher, overlong reward shaping, and maintaining adequate policy entropy could improve the training efficiency.
  arXiv  Detail & Related papers  (2025-10-13T17:57:15Z)
• RL in the Wild: Characterizing RLVR Training in LLM Deployment [43.81962834561768]
  Reinforcement Learning with Verifiable Rewards (RLVR) has surged in recent months to enhance their reasoning and understanding abilities.<n>However, its complex data flows and diverse tasks pose substantial challenges to RL training systems.<n>There is limited understanding of RLVR from a system perspective.
  arXiv  Detail & Related papers  (2025-09-29T03:09:27Z)
• AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning [129.44038804430542]
  We introduce AgentGym-RL, a new framework to train LLM agents for multi-turn interactive decision-making through RL.<n>We propose ScalingInter-RL, a training approach designed for exploration-exploitation balance and stable RL optimization.<n>Our agents match or surpass commercial models on 27 tasks across diverse environments.
  arXiv  Detail & Related papers  (2025-09-10T16:46:11Z)
• rStar2-Agent: Agentic Reasoning Technical Report [25.266747156205266]
  We introduce rStar2-Agent, a 14B math reasoning model trained with agentic reinforcement learning to achieve frontier-level performance.<n>To this end, rStar2-Agent boosts a pre-trained 14B model to state of the art in only 510 RL steps within one week, achieving average pass@1 scores of 80.6% on AIME24 and 69.8% on AIME25.
  arXiv  Detail & Related papers  (2025-08-28T12:45:25Z)
• LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback [121.78866929908871]
  Large Action Models (LAMs) for AI Agents offer incredible potential but face challenges due to the need for high-quality training data.<n>We present LAM SIMULATOR, a comprehensive framework designed for online exploration of agentic tasks with high-quality feedback.<n>Our framework features a dynamic task query generator, an extensive collection of tools, and an interactive environment where Large Language Model (LLM) Agents can call tools and receive real-time feedback.
  arXiv  Detail & Related papers  (2025-06-02T22:36:02Z)
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning [125.96848846966087]
  Training large language models (LLMs) as interactive agents presents unique challenges.<n>While reinforcement learning has enabled progress in static tasks, multi-turn agent RL training remains underexplored.<n>We propose StarPO, a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents.
  arXiv  Detail & Related papers  (2025-04-24T17:57:08Z)
• Contrastive Representation for Interactive Recommendation [20.020630759453237]
  We propose Contrastive Representation for Interactive Recommendation (CRIR)<n>CRIR efficiently extracts latent, high-level preference ranking features from explicit interaction.<n>We also propose a data exploiting mechanism and an agent training mechanism to better adapt CRIR to the Deep Reinforcement Learning backbone.
  arXiv  Detail & Related papers  (2024-12-24T12:39:23Z)
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning [61.10299147201369]
  This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents. We build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement.
  arXiv  Detail & Related papers  (2024-06-14T17:49:55Z)
• TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents [10.798271657186492]
  This paper advocates Trajdeleter, the first practical approach to trajectory unlearning for offline RL agents. The key idea of Trajdeleter is to guide the agent to demonstrate deteriorating performance when it encounters states associated with unlearning trajectories. Extensive experiments conducted on six offline RL algorithms and three tasks demonstrate that Trajdeleter requires only about 1.5% of the time needed for retraining from scratch.
  arXiv  Detail & Related papers  (2024-04-18T22:23:24Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)
• Retrieval-Augmented Reinforcement Learning [63.32076191982944]
  We train a network to map a dataset of past experiences to optimal behavior. The retrieval process is trained to retrieve information from the dataset that may be useful in the current context. We show that retrieval-augmented R2D2 learns significantly faster than the baseline R2D2 agent and achieves higher scores.
  arXiv  Detail & Related papers  (2022-02-17T02:44:05Z)
• Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning [85.86440477005523]
  We study more human-like RL agents which incorporate an established model of human-irrationality, the Rational Inattention (RI) model. RIRL models the cost of cognitive information processing using mutual information. We show that using RIRL yields a rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions.
  arXiv  Detail & Related papers  (2022-01-18T20:54:00Z)
• Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving [5.404179497338455]
  Recent advances in Reinforcement Learning (RL) combined with Deep Learning (DL) have demonstrated impressive performance in complex tasks, including autonomous driving. We propose HPRL - Hierarchical Program-triggered Reinforcement Learning, which uses a hierarchy consisting of a structured program along with multiple RL agents, each trained to perform a relatively simple task. The focus of verification shifts to the master program under simple guarantees from the RL agents, leading to a significantly more interpretable and verifiable implementation as compared to a complex RL agent.
  arXiv  Detail & Related papers  (2021-03-25T14:19:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.