Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams

• URL: http://arxiv.org/abs/2606.20629v1
• Date: Thu, 28 May 2026 17:12:20 GMT
• Title: Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams
• Abstract summary: LLM agents are increasingly deployed as multi-role teams, where tasks are divided across specialized roles such as planner, executor, and verifier.<n>Existing agentic benchmarks evaluate fixed models or fixed agent configurations, and offer limited guidance for cost-accuracy-optimal deployment.<n>We introduce AgentCARD, a role-aware benchmark suite for evaluating LLM agent teams across role assignment and deployment mode.
• Score: 14.784600350594614
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: LLM agents are increasingly deployed as multi-role teams, where tasks are divided across specialized roles such as planner, executor, and verifier. In these systems, cost and accuracy are no longer properties of a single model: they depend on which model fills each role and where it is hosted, including API, self-hosted, and hybrid deployment. Existing agentic benchmarks typically evaluate fixed models or fixed agent configurations, and therefore offer limited guidance for cost-accuracy-optimal deployment. We introduce AgentCARD, a role-aware benchmark suite for evaluating LLM agent teams across role assignment and deployment mode. AgentCARD combines a role-decomposed evaluation harness, a unified API/self-hosted cost model, Pareto-frontier analysis, and a Shapley-based diagnostic for identifying role bottlenecks. Our evaluation shows that heterogeneous teams consistently occupy the cost-accuracy frontier. They improve accuracy by up to $44\%$ over cost-equivalent homogeneous teams, or match the strongest homogeneous team at up to $12\times$ lower per-task cost through hybrid deployment. We further find that the best role assignment is domain-dependent: some domains are planner-bottlenecked, while others are executor-bottlenecked. Finally, AgentCARD extends beyond planner--executor teams to workflows with additional roles such as verification, and supports continual evaluation as new domains and team structures emerge. Our code is released at: https://github.com/Auto-CAP/AgentCAP

Related papers

• Markets, Not Planners: Decentralized Orchestration of LLM Agents with Private Information [16.55988307561715]
  We introduce AgentLance, a labor market in which agents bid on tasks using their private costs and self-maintained strategy notes.<n>AgentLance consistently outperforms single-model, centralized-orchestration, and market baselines.
  arXiv  Detail & Related papers  (2026-08-24T22:15:37Z)
• AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints [0.0]
  AgentSLABench is a resource-aware evaluation framework for autonomous AI agents.<n>It measures correctness alongside latency, cost, compute, memory, and network usage under declared resource budgets.
  arXiv  Detail & Related papers  (2026-08-01T18:17:47Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents [0.0]
  CFAgentBench is a benchmark for autonomous construction-finance agents.<n>It contains 1,014 machine-gradeable task specifications across 8 domains and 77 families.<n>The strongest agent reaches pass1 = 0.67 but only pass5 = 0.38.
  arXiv  Detail & Related papers  (2026-06-20T11:34:52Z)
• Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [48.1363632826625]
  Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world.<n>We introduce $alem$, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics.
  arXiv  Detail & Related papers  (2026-06-06T21:13:43Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent [18.58248061903799]
  We introduce AgentOpt, the first framework-agnostic Python package for client-side AI optimization.<n>We first study model selection, a high-impact optimization lever in multi-step agent pipelines.<n>To efficiently explore the exponentially growing combination space, AgentOpt implements ten search algorithms.
  arXiv  Detail & Related papers  (2026-04-07T17:13:47Z)
• Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO [24.532870400949424]
  Current training methods train a unified large language model for all agents in the system.<n>This may limit the performances due to different underlying distributions for different agents.<n>We propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization for vertical Multi-agent systems.
  arXiv  Detail & Related papers  (2025-11-17T12:06:30Z)
• AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress [71.02263260394261]
  Large language models (LLMs) still encounter challenges in multi-turn decision-making tasks.<n>We build process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process.<n>AgentPRM captures both the interdependence between sequential decisions and their contribution to the final goal.
  arXiv  Detail & Related papers  (2025-11-11T14:57:54Z)
• Multi-Agent Tool-Integrated Policy Optimization [67.12841355267678]
  Large language models (LLMs) increasingly rely on multi-turn tool-integrated planning for knowledge-intensive and complex reasoning tasks.<n>Existing implementations typically rely on a single agent, but they suffer from limited context length and noisy tool responses.<n>No existing methods support effective reinforcement learning post-training of tool-integrated multi-agent frameworks.
  arXiv  Detail & Related papers  (2025-10-06T10:44:04Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents.<n>The impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored.<n>This paper investigates what is the resilience of various system structures under faulty agents on different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)
• A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration [55.35849138235116]
  We propose automatically selecting a team of agents from candidates to collaborate in a dynamic communication structure toward different tasks and domains. Specifically, we build a framework named Dynamic LLM-Powered Agent Network ($textDyLAN$) for LLM-powered agent collaboration. We demonstrate that DyLAN outperforms strong baselines in code generation, decision-making, general reasoning, and arithmetic reasoning tasks with moderate computational cost.
  arXiv  Detail & Related papers  (2023-10-03T16:05:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.