Reward Modeling for Multi-Agent Orchestration

• URL: http://arxiv.org/abs/2606.13598v1
• Date: Thu, 11 Jun 2026 17:16:24 GMT
• Title: Reward Modeling for Multi-Agent Orchestration
• Abstract summary: Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents.<n>We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations.
• Score: 43.79233309038303
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents, yet training such orchestrators is hindered by limited supervision and high computational cost. We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations. OrchRM leverages intermediate artifacts from multi-agent executions to construct win-lose pairs for Bradley-Terry reward model training. Unlike existing MAS test-time scaling and orchestrator training frameworks that rely on costly sub-agent rollouts, OrchRM operates directly at the orchestration level, enabling efficient and high-performing reward-guided orchestrator training and MAS test-time scaling. OrchRM improves training efficiency by up to 10x in token usage while improving MAS test-time scaling performance by up to 8% in accuracy. These gains consistently transfer across multiple domains, including mathematical reasoning, web-based question answering, and multi-hop reasoning, demonstrating orchestration-level reward modeling as a scalable direction for robust multi-agent orchestration. Code will be available at https://github.com/Wang-ML-Lab/OrchRM.

Related papers

• SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks [65.14594927081983]
  We present SciOrch, a framework that trains a lightweight 8B model to orchestrate frontier LLMs for scientific reasoning.<n>On a 240-question test set spanning SGI-Reasoning and Scientists' First exam, SciOrch reaches 56.66% average accuracy.<n>It also attains the best accuracy on both SGI and SFE with less than half the API cost of typical multi-agent methods.
  arXiv  Detail & Related papers  (2026-06-14T15:45:34Z)
• LEMON: Learning Executable Multi-Agent Orchestration via Counterfactual Reinforcement Learning [31.870185345616733]
  Large language models (LLMs) have become a strong foundation for multi-agent systems, but their effectiveness depends heavily on orchestration design.<n>We propose LEMON, an orchestrator that generates an executable orchestration specification.<n> Experiments on six reasoning and coding benchmarks, including MMLU, GSM8K, AQuA, MultiArith, SVAMP, and HumanEval, show that LEMON achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-05-14T07:24:09Z)
• SkillOrchestra: Learning to Route Agents via Skill Transfer [65.50924963973286]
  We introduce SkillOrchestra, a framework for skill-aware orchestration.<n>SkillOrchestra learns fine-grained skills from execution experience and models agent-specific competence and cost under those skills.<n>At deployment, the orchestrator infers the skill demands of the current interaction and selects agents that best satisfy them under an explicit performance-cost trade-off.
  arXiv  Detail & Related papers  (2026-02-23T10:17:25Z)
• MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks [86.05918381895555]
  We propose MASOrchestra as a training-time framework that formulates MAS orchestration as a function-calling reinforcement learning problem.<n>In MAS-Orchestra, complex, goal-oriented subagents are abstracted as callable functions, enabling global reasoning over system structure.<n>Our analysis reveals that MAS gains depend critically on task structure, verification protocols, and the capabilities of both orchestrator and subagents.
  arXiv  Detail & Related papers  (2026-01-21T04:57:02Z)
• Multi-Agent Collaboration via Evolving Orchestration [55.574417128944226]
  Large language models (LLMs) have achieved remarkable results across diverse downstream tasks, but their monolithic nature restricts scalability and efficiency in complex problem-solving.<n>We propose a puppeteer-style paradigm for LLM-based multi-agent collaboration, where a centralized orchestrator ("puppeteer") dynamically directs agents ("puppets") in response to evolving task states.<n> Experiments on closed- and open-domain scenarios show that this method achieves superior performance with reduced computational costs.
  arXiv  Detail & Related papers  (2025-05-26T07:02:17Z)
• Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark [72.46357004059661]
  Generalist Virtual Agents (GVAs) have shown significant promise in autonomous task execution.<n>To address these challenges, we propose Similar, a Step-Wise Multi-Dimensional Generalist Reward Model.<n>Similar offers fine-grained signals for agent training and can choose better action for inference-time scaling.
  arXiv  Detail & Related papers  (2025-03-24T13:30:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.