Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces

• URL: http://arxiv.org/abs/2605.02801v1
• Date: Mon, 04 May 2026 16:42:18 GMT
• Title: Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces
• Abstract summary: Large language model (LLM) agents evolve from isolated tool users into coordinated teams.<n> reinforcement learning must optimize not only individual actions but also how work is spawned, delegated, communicated, aggregated, and stopped.<n>This paper studies reinforcement learning for multi-agent systems through orchestration traces.
• Score: 3.651838585096184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language model (LLM) agents evolve from isolated tool users into coordinated teams, reinforcement learning (RL) must optimize not only individual actions but also how work is spawned, delegated, communicated, aggregated, and stopped. This paper studies RL for LLM-based multi-agent systems through orchestration traces: temporal interaction graphs whose events include sub-agent spawning, delegation, communication, tool use, return, aggregation, and stopping decisions. Using this lens, we identify three technical axes. First, reward design spans eight families, including orchestration rewards for parallelism speedup, split correctness, and aggregation quality. Second, reward and credit signals attach to eight credit- or signal-bearing units from token to team; explicit counterfactual message-level credit remains especially sparse in our curated pool. Third, orchestration learning decomposes into five sub-decisions: when to spawn, whom to delegate to, how to communicate, how to aggregate, and when to stop. In our curated pool as of May 4, 2026, we found no explicit RL training method for the stopping decision. We connect academic methods to public industrial evidence from Kimi Agent Swarm, OpenAI Codex, and Anthropic Claude Code. The resulting scale gap is a gap between publicly reported deployment envelopes and open academic evaluation regimes, not independent verification of industrial training traces. We release the artifact at https://github.com/xxzcc/awesome-llm-mas-rl, including an 84-entry tagged paper pool, a 32-record exclusion log, scripted corpus statistics, and a minimal JSON schema for replayable orchestration traces.

Related papers

• Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents [3.3656546156010942]
  Reinforcement learning is a natural way to post-train agents for long-horizon interactive tasks judged only by end-of-task verification.<n>Recent work compensates around the training with denser rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration.<n>We present CANOPY, a minimalist protocol attacking both directly: scale same-task exploration until the natural signal reappears, keep every update on-policy, KL-anchored, and confined to the agent's own action tokens.
  arXiv  Detail & Related papers  (2026-09-01T13:44:40Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• LEMON: Learning Executable Multi-Agent Orchestration via Counterfactual Reinforcement Learning [31.870185345616733]
  Large language models (LLMs) have become a strong foundation for multi-agent systems, but their effectiveness depends heavily on orchestration design.<n>We propose LEMON, an orchestrator that generates an executable orchestration specification.<n> Experiments on six reasoning and coding benchmarks, including MMLU, GSM8K, AQuA, MultiArith, SVAMP, and HumanEval, show that LEMON achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-05-14T07:24:09Z)
• When LLMs Team Up: A Coordinated Attack Framework for Automated Cyber Intrusions [27.62993988829038]
  This paper presents CAESAR, a coordinated multi-agent framework for analysis of intrusion-style tasks.<n>CAESAR decomposes the workflow into five typed roles and coordinates them through a bounded protocol.<n>Compared with a single-agent baseline under matched budgets and tool access, CAESAR improves task success and reduces performance variance.
  arXiv  Detail & Related papers  (2026-05-09T07:47:42Z)
• Bian Que: An Agentic Framework with Flexible Skill Arrangement for Online System Operations [19.829321356625428]
  Bian Que is an agentic framework for monitoring, alert response, and root cause analysis.<n>It was deployed on the e-commerce search engine of KuaiShou, the major short-video platform in China.<n>It reduces alert volume by 75%, achieves 80% root-cause analysis accuracy, and cuts mean time to resolution by over 50%.
  arXiv  Detail & Related papers  (2026-04-29T15:35:01Z)
• Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards [76.49428173793386]
  LLMs must invoke multiple dependent APIs in the correct order while propagating intermediate outputs.<n>Training models to handle such faces two obstacles: existing environments focus on simple per-turn function calls with simulated data, and binary rewards.<n>We present a framework addressing both challenges. First, we construct a reinforcement learning environment backed by a large-scale cache of real API responses, enabling a data synthesis pipeline that samples valid multi-step orchestration traces.<n>Second, we propose a graduated reward design that decomposes correctness into atomic validity and orchestration.
  arXiv  Detail & Related papers  (2026-03-25T18:31:39Z)
• OpenClaw-RL: Train Any Agent Simply by Talking [54.06773485601523]
  Next-state signals are universal, and policy can learn from all of them simultaneously.<n>Personal conversations, terminal executions, GUI interactions, SWE tasks, and tool-call traces are not separate training problems.<n>OpenClaw-RL enables an agent to improve simply by being used, recovering conversational signals from user re-queries, corrections, and explicit feedback.
  arXiv  Detail & Related papers  (2026-03-10T18:59:01Z)
• SCoUT: Scalable Communication via Utility-Guided Temporal Grouping in Multi-Agent Reinforcement Learning [2.064923532131528]
  Communication can improve coordination in partially observed multi-agent reinforcement learning (MARL)<n>We introduce textbfSCoUT (textbfScalable textbfCommunication via textbfUtility-guided textbfTemporal grouping)<n>Agents are trained with a three-headed policy: environment action, send decision, and recipient selection.
  arXiv  Detail & Related papers  (2026-03-05T05:33:28Z)
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning [125.96848846966087]
  Training large language models (LLMs) as interactive agents presents unique challenges.<n>While reinforcement learning has enabled progress in static tasks, multi-turn agent RL training remains underexplored.<n>We propose StarPO, a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents.
  arXiv  Detail & Related papers  (2025-04-24T17:57:08Z)
• MALT: Improving Reasoning with Multi-Agent LLM Training [67.76186488361685]
  MALT (Multi-Agent LLM Training) is a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps.<n>On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively.
  arXiv  Detail & Related papers  (2024-12-02T19:30:36Z)
• Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback [52.763620660061115]
  ONI is a distributed architecture that simultaneously learns an RL policy and an intrinsic reward function.<n>We explore a range of algorithmic choices for reward modeling with varying complexity.<n>Our approach achieves state-of-the-art performance across a range of challenging tasks from the NetHack Learning Environment.
  arXiv  Detail & Related papers  (2024-10-30T13:52:43Z)
• Efficient Parallel Reinforcement Learning Framework using the Reactor Model [2.190190313041532]
  Reinforcement Learning (RL) frameworks are essential for mapping RL workloads to multiple computational resources. Existing frameworks, such as Ray, are not managing this orchestration efficiently. We have proposed a solution implementing the reactor model, which enforces a set of actors to have a fixed communication pattern.
  arXiv  Detail & Related papers  (2023-12-07T21:19:57Z)
• Decentralized Cooperative Multi-Agent Reinforcement Learning with Exploration [35.75029940279768]
  We study multi-agent reinforcement learning in the most basic cooperative setting -- Markov teams. We propose an algorithm in which each agent independently runs a stage-based V-learning style algorithm. We show that the agents can learn an $epsilon$-approximate Nash equilibrium policy in at most $proptowidetildeO (1/epsilon4)$ episodes.
  arXiv  Detail & Related papers  (2021-10-12T02:45:12Z)
• MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning [61.28547338576706]
  Population-based multi-agent reinforcement learning (PB-MARL) refers to the series of methods nested with reinforcement learning (RL) algorithms. We present MALib, a scalable and efficient computing framework for PB-MARL.
  arXiv  Detail & Related papers  (2021-06-05T03:27:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.