Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis

• URL: http://arxiv.org/abs/2604.14877v1
• Date: Thu, 16 Apr 2026 11:06:19 GMT
• Title: Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis
• Abstract summary: We introduce PASS@(k,T), a metric that separates capability expansion from efficiency improvement.<n>Our main finding is that, contrary to the static-reasoning result, tool-use RL genuinely enlarges the capability boundary.<n>Under matched training data, supervised fine-tuning regresses the boundary on the same compositional tasks, isolating self-directed exploration as the causal factor.
• Score: 16.739617199649615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Does reinforcement learning genuinely expand what LLM agents can do, or merely make them more reliable? For static reasoning, recent work answers the second: base and RL pass@k curves converge at large k. We ask whether this holds for agentic tool use, where T rounds of interaction enable compositional strategies that re-sampling cannot recover. We introduce PASS@(k,T), a two-dimensional metric that jointly varies sampling budget k and interaction depth T, separating capability expansion from efficiency improvement. Our main finding is that, contrary to the static-reasoning result, tool-use RL genuinely enlarges the capability boundary: the RL agent's pass-curve pulls above the base model's and the gap widens at large k rather than converging. The expansion is specific to compositional, sequential information gathering; on simpler tasks RL behaves as prior work predicts. Under matched training data, supervised fine-tuning regresses the boundary on the same compositional tasks, isolating self-directed exploration as the causal factor. Mechanism analysis shows RL reweights the base strategy distribution toward the subset whose downstream reasoning more often yields a correct answer, with the improvement concentrated on how the agent integrates retrieved information. These results reconcile optimistic and pessimistic readings of RL for LLMs: both are correct, on different task types.

Related papers

• Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL [55.05743310621117]
  Co-RL is a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers.<n>We show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops.
  arXiv  Detail & Related papers  (2026-08-18T01:16:02Z)
• ExpRL: Exploratory RL for LLM Mid-Training [40.4311030968937]
  Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning.<n>We study a more automated approach: emphRL-based mid-training using large corpora of human-written question-answer data.<n>Our method, ExpRL, uses them as emphreward scaffolds: references are hidden from the policy and used only to construct problem-specific grading rubrics.
  arXiv  Detail & Related papers  (2026-06-15T17:50:44Z)
• RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training [18.932290080075685]
  We find that RL is effective very early, and often matches the full SFT$to$RL pipeline early as well.<n>Applying RL directly to base checkpoints expands the model's distribution.<n>The general capabilities of the model remain essentially unchanged by RL, while they degrade following SFT.
  arXiv  Detail & Related papers  (2026-06-02T22:55:18Z)
• DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
  We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
  arXiv  Detail & Related papers  (2026-05-17T07:14:44Z)
• Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs [59.62441340813425]
  We study whether reinforcement learning can improve direct recall of parametric knowledge.<n>Across three model families and multiple factual QA benchmarks, RL yields 27% average relative gains.<n>Mechanistically, RL primarily redistributes probability mass over existing knowledge rather than acquiring new facts.
  arXiv  Detail & Related papers  (2026-05-08T02:40:12Z)
• MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance [18.215893951726166]
  Large language models (LLMs) can provide subgoal decompositions, plausible trajectories, and abstract priors that facilitate early learning.<n>We propose MIRA (Memory-Integrated Reinforcement Learning Agent), which incorporates a structured, evolving memory graph to guide early training.
  arXiv  Detail & Related papers  (2026-02-20T01:43:30Z)
• From Supervision to Exploration: What Does Protein Language Model Learn During Reinforcement Learning? [76.288870982181]
  Protein language models (PLMs) have advanced computational protein science through large-scale pretraining and scalable architectures.<n> reinforcement learning (RL) has broadened exploration and enabled precise multi-objective optimization in protein design.<n>We ask if RL improves sampling efficiency and, more importantly, if it reveals capabilities not captured by supervised learning.
  arXiv  Detail & Related papers  (2025-10-02T01:31:10Z)
• From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones [68.68686526804909]
  We show that LLMs can acquire genuinely new skills during RL by composing existing ones.<n>Our experiments show that compositional skill acquired on a source task transfers to a different target task.<n>This transfer happens even without compositional training on the target, requiring only prior knowledge of the target's atomic skills.
  arXiv  Detail & Related papers  (2025-09-29T17:44:27Z)
• RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization [111.1749164063616]
  We propose RL-PLUS, a novel hybrid-policy optimization approach for Large Language Models (LLMs)<n> RL-PLUS synergizes internal exploitation with external data to achieve stronger reasoning capabilities and surpass the boundaries of base models.<n>We provide both theoretical analysis and extensive experiments to demonstrate the superiority and generalizability of our approach.
  arXiv  Detail & Related papers  (2025-07-31T23:55:29Z)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning [93.00629872970364]
  Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks.<n>We introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions.<n>We study whether difficult problems -- those yielding no RL signals and mixed-quality reasoning traces -- can still be effectively used for training.
  arXiv  Detail & Related papers  (2025-06-05T07:53:59Z)
• Decomposing Elements of Problem Solving: What "Math" Does RL Teach? [22.517954679764244]
  We decompose problem solving into fundamental capabilities: Plan, Execute, and Verify.<n>We show that RL-trained models struggle with fundamentally new problems, hitting a 'coverage wall' due to insufficient planning skills.<n>Our findings provide insights into the role of RL in enhancing LLM reasoning, expose key limitations, and suggest a path toward overcoming these barriers.
  arXiv  Detail & Related papers  (2025-05-28T18:18:49Z)
• Teaching Large Language Models to Reason with Reinforcement Learning [38.17625148525193]
  Reinforcement Learning from Human Feedback (textbfRLHF) has emerged as a dominant approach for aligning LLM outputs with human preferences. Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback.
  arXiv  Detail & Related papers  (2024-03-07T16:36:29Z)
• More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning [58.626683114119906]
  We show that Distributional Reinforcement Learning (DistRL) can obtain second-order bounds in both online and offline RL. Our results are the first second-order bounds for low-rank MDPs and for offline RL.
  arXiv  Detail & Related papers  (2024-02-11T13:25:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.