Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

• URL: http://arxiv.org/abs/2604.10547v1
• Date: Sun, 12 Apr 2026 09:35:27 GMT
• Title: Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
• Abstract summary: Agent2 RL-Bench is a benchmark for evaluating agentic RL post-training.<n>We find that supervised pipelines dominate agent-driven post-training under fixed budgets.
• Score: 23.212983812736343
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce Agent^2 RL-Bench, a benchmark for evaluating agentic RL post-training -- whether LLM agents can autonomously design, implement, and run complete RL pipelines that improve foundation models. This capability is important because RL post-training increasingly drives model alignment and specialization, yet existing benchmarks remain largely static: supervised fine-tuning alone yields strong results, leaving interactive RL engineering untested. Agent^2 RL-Bench addresses this with six tasks across three levels -- from static rule-based training to closed-loop online RL with trajectory collection -- each adding a structural requirement that prior levels do not impose. The benchmark provides isolated workspaces with a grading API, runtime instrumentation that records every submission and code revision, and automated post-hoc analysis that generates structured run reports, enabling the first automated diagnostic of agent-driven post-training behavior. Across multiple agent stacks spanning five agent systems and six driver LLMs, we find that agents achieve striking interactive gains -- on ALFWorld, an RL-only agent improves from 5.97 to 93.28 via SFT warm-up and GRPO with online rollouts -- yet make only marginal progress on others (DeepSearchQA: +2.75 within evaluation noise), and that driver choice has a large effect on interactive tasks -- within the same scaffold, switching drivers changes interactive improvement from near-zero to +78pp. More broadly, the benchmark reveals that supervised pipelines dominate agent-driven post-training under fixed budgets, with online RL succeeding as the final best route only on ALFWorld. Code is available at https://github.com/microsoft/RD-Agent/tree/main/rdagent/scenarios/rl/autorl_bench.

Related papers

• Agent Lightning v1.0: Towards Harnessed Agentic RL [28.087691629023038]
  We present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL.<n>It supports arbitrary agent harnesses and serves as a practical testbed for studying these challenges.<n>We evaluate it on instruction-following, search, and coding agents, and provide a complete reproducible pipeline for coding-agent RL.
  arXiv  Detail & Related papers  (2026-08-18T08:50:13Z)
• ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents [75.40987300040632]
  ProRL Agent is a scalable infrastructure that serves the full agentic rollout lifecycle through an API service.<n>ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym.
  arXiv  Detail & Related papers  (2026-03-19T12:08:51Z)
• Behavior Knowledge Merge in Reinforced Agentic Models [48.89546963456286]
  Reinforcement learning is central to post-training, particularly for agentic models that require specialized reasoning behaviors.<n>Existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models.<n>We propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models.
  arXiv  Detail & Related papers  (2026-01-20T03:56:53Z)
• AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework [76.96794548655292]
  Large language models (LLMs) have sparked growing interest in building generalist agents that can learn through online interactions.<n>Applying reinforcement learning (RL) to train LLM agents in multi-turn, multi-task settings remains challenging due to lack of scalable infrastructure and stable training algorithms.<n>We present the AgentRL framework for scalable multi-turn, multi-task agentic RL training.
  arXiv  Detail & Related papers  (2025-10-05T13:40:01Z)
• $Agent^2$: An Agent-Generates-Agent Framework for Reinforcement Learning Automation [5.325886106098561]
  Reinforcement learning (RL) agent development traditionally requires substantial expertise and iterative effort.<n>This paper introduces Agent$2$, an LLM-driven agent-generates-agent framework for fully automated RL agent design.<n>Agent$2$ translates natural language task descriptions and environment code into executable RL solutions without human intervention.
  arXiv  Detail & Related papers  (2025-09-16T02:14:39Z)
• Agent Lightning: Train ANY AI Agents with Reinforcement Learning [24.13422767414729]
  We present Agent Lightning, a framework that enables Reinforcement Learning (RL)-based training of Large Language Models (LLMs) for any AI agent.<n>By formulating agent execution as Markov decision process, we define an unified data interface and propose a hierarchical RL algorithm, LightningRL, which contains a credit assignment module.<n>For the system design, we introduce a Training-Agent Disaggregation architecture, and brings agent observability frameworks into agent runtime.
  arXiv  Detail & Related papers  (2025-08-05T17:50:13Z)
• MALT: Improving Reasoning with Multi-Agent LLM Training [67.76186488361685]
  MALT (Multi-Agent LLM Training) is a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps.<n>On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively.
  arXiv  Detail & Related papers  (2024-12-02T19:30:36Z)
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning [61.10299147201369]
  This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents. We build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement.
  arXiv  Detail & Related papers  (2024-06-14T17:49:55Z)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL [80.10358123795946]
  We develop a framework for building multi-turn RL algorithms for fine-tuning large language models. Our framework adopts a hierarchical RL approach and runs two RL algorithms in parallel. Empirically, we find that ArCHer significantly improves efficiency and performance on agent tasks.
  arXiv  Detail & Related papers  (2024-02-29T18:45:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.