MCP-Universe RL: A Framework for Training MCP Tool-Use Agents via Reinforcement Learning

• URL: http://arxiv.org/abs/2608.22167v1
• Date: Sun, 23 Aug 2026 01:49:04 GMT
• Title: MCP-Universe RL: A Framework for Training MCP Tool-Use Agents via Reinforcement Learning
• Abstract summary: Reinforcement learning (RL) has become an effective way to improve the tool-use ability of large language models (LLMs)<n>Most existing RL frameworks stop at the policy update.<n>We present MCP-Universe RL (MCP-U RL), an open-source framework that takes over both.
• Score: 54.082979654934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) has become an effective way to improve the tool-use ability of large language models (LLMs), but most existing RL frameworks stop at the policy update. For every new domain, the user is left with two hard systems problems: standing up an isolated environment for each of hundreds of concurrent trajectories and connecting it to training, and scheduling the rollout so that the GPU stays busy across long, multi-turn episodes that spend much of their time stalled on slow tool calls. We present MCP-Universe RL (MCP-U RL), an open-source framework that takes over both. It uses the Model Context Protocol (MCP) as the interface to the environment, so any tool already exposed as an MCP server plugs into training with no RL-specific integration code. It builds the two missing layers once and reuses them across domains: an environment-orchestration layer that provisions, isolates, and recycles the MCP environments over a pluggable container backend, and a rollout-orchestration layer whose staged pipeline overlaps trajectories to keep the GPU busy while episodes wait on tools. A backend-agnostic training layer then applies the update through an existing RL backend, with veRL and slime integrations. With one configuration, changing only the task specification, we train software-engineering, deep-research, and general tool-use agents on gpt-oss-20b and improve task reward in all three.

Related papers

• ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents [75.40987300040632]
  ProRL Agent is a scalable infrastructure that serves the full agentic rollout lifecycle through an API service.<n>ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym.
  arXiv  Detail & Related papers  (2026-03-19T12:08:51Z)
• RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure [49.88201789074532]
  Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning.<n>We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure.
  arXiv  Detail & Related papers  (2025-12-27T11:14:23Z)
• Role-Based Fault Tolerance System for LLM RL Post-Training [24.163081819380647]
  RL post-training interleaves training and inference workloads, exposing the system to faults from both sides.<n>Our key insight is role-based fault isolation so the failure in one machine does not affect the others.<n>We present RobustRL, the first comprehensive robust system to handle GPU machine errors for RL post-training Effective Training Time Ratio improvement.
  arXiv  Detail & Related papers  (2025-12-27T06:30:18Z)
• Laminar: A Scalable Asynchronous RL Post-Training Framework [20.127034898123508]
  Long-tail skewness in RL trajectory generation causes severe GPU underutilization.<n>Current RL systems rely on global weight synchronization between the actor and all rollouts, which creates a rigid model update schedule.<n>We propose Laminar, a scalable and robust RL post-training system built on a fully decoupled architecture.
  arXiv  Detail & Related papers  (2025-10-14T15:29:14Z)
• AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework [76.96794548655292]
  Large language models (LLMs) have sparked growing interest in building generalist agents that can learn through online interactions.<n>Applying reinforcement learning (RL) to train LLM agents in multi-turn, multi-task settings remains challenging due to lack of scalable infrastructure and stable training algorithms.<n>We present the AgentRL framework for scalable multi-turn, multi-task agentic RL training.
  arXiv  Detail & Related papers  (2025-10-05T13:40:01Z)
• RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use [50.52940111891476]
  Large language models excel at basic reasoning but struggle with tasks that require interaction with external tools.<n>We present RLFactory, a plug-and-play reinforcement learning framework for multi-round tool use.
  arXiv  Detail & Related papers  (2025-08-31T16:47:31Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs)<n>StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks.<n> Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)
• RLtools: A Fast, Portable Deep Reinforcement Learning Library for Continuous Control [7.259696592534715]
  Deep Reinforcement Learning (RL) can yield capable agents and control policies in several domains but is commonly plagued by prohibitively long training times. We present RLtools, a dependency-free, header-only, pure C++ library for deep supervised and reinforcement learning.
  arXiv  Detail & Related papers  (2023-06-06T09:26:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.