Rethinking the Evaluation of Efficiency Methods for Multi-Agent Systems

• URL: http://arxiv.org/abs/2609.05933v1
• Date: Sat, 05 Sep 2026 06:47:38 GMT
• Title: Rethinking the Evaluation of Efficiency Methods for Multi-Agent Systems
• Abstract summary: Recent methods aim to make Large Language Model (LLM)-based multi-agent systems cheaper by pruning agents, removing communication edges, or searching for compact structures.<n>We argue that existing evaluations may overestimate their true ability to improve MAS efficiency.<n>Our analysis shows that many reported gains are setup-dependent and may arise from structural collapse, disabled tool pathways, or starting systems where random pruning already preserves accuracy, rather than robust improvements in MAS efficiency.
• Score: 22.50944765658515
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Efficiency is increasingly important for Large Language Model (LLM)-based multi-agent systems (MAS), as larger models and more agents introduce substantial execution costs. Recent methods aim to make MAS cheaper by pruning agents, removing communication edges, or searching for compact structures. However, we argue that existing evaluations may overestimate their true ability to improve MAS efficiency. Reported gains are often measured under method-specific prompts and starting topologies, making them difficult to attribute to the proposed structural changes. Moreover, many reported successes appear in non-MAS-demanding settings, where a single agent or a randomly pruned system can already preserve strong performance. To study these issues, we introduce a controlled and MAS-demanding diagnostic benchmark for representative MAS efficiency methods. We evaluate methods under a shared backbone model, agent registry, and runtime, across controlled variations in topology, scale, depth, and tool use. Our analysis shows that many reported gains are setup-dependent and may arise from structural collapse, disabled tool pathways, or starting systems where random pruning already preserves accuracy, rather than robust improvements in MAS efficiency.

Related papers

• Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attribution [47.70578021328887]
  Large language model (LLM)-based multi-agent systems often exhibit complex failure modes.<n>We introduce AFANet, a lightweight graph-based framework that models interaction trajectories through step-level semantic signals and agent-level relationships.<n>Our results suggest that effective agent failure attribution does not require heavy reasoning and a lightweight, structured approach can achieve strong performance.
  arXiv  Detail & Related papers  (2026-08-19T06:16:09Z)
• The Optimizer Is the Agent: Reasoning-Driven Search across Prompts, Programs, and ML Workflows [72.9051182568928]
  We present ReASearch, a unified framework for reasoning-driven optimization.<n>With a shared agent loop and domain-specific tools, ReASearch instantiates the exact same scaffold to optimize prompts, programs, and ML.<n>We observe that complex search behaviors emerge naturally from the agent's reasoning process.
  arXiv  Detail & Related papers  (2026-08-07T02:15:39Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets [14.028807574426748]
  We present an information-theoretic argument, grounded in the Data Processing Inequality.<n>We find that SAS consistently match or outperform MAS on multi-hop reasoning tasks when reasoning tokens are held constant.<n>Overall, our results suggest that, for multi-hop reasoning tasks, many reported advantages of multi-agent systems are better explained by unaccounted computation and context effects.
  arXiv  Detail & Related papers  (2026-04-02T18:47:48Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• A Systematic Study of LLM-Based Architectures for Automated Patching [7.9821766277253845]
  We present a controlled evaluation of four large language models (LLMs)-based patching paradigms.<n>We analyze patch correctness, failure modes, token usage, and execution time across real-world vulnerability tasks.<n>Our results reveal clear architectural trade-offs: fixed are efficient but brittle, single-agent systems balance flexibility and cost, and multi-agent designs improve at the expense of substantially higher overhead.
  arXiv  Detail & Related papers  (2026-03-01T20:26:22Z)
• AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent [57.10083973844841]
  AgentArk is a novel framework to distill multi-agent dynamics into the weights of a single model.<n>We investigate three hierarchical distillation strategies across various models, tasks, scaling, and scenarios.<n>By shifting the burden of computation from inference to training, the distilled models preserve the efficiency of one agent while exhibiting strong reasoning and self-correction performance of multiple agents.
  arXiv  Detail & Related papers  (2026-02-03T19:18:28Z)
• MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems [59.20800753428596]
  We present MAS-ProVe, a systematic empirical study of process verification for multi-agent systems (MAS)<n>Our study spans three verification paradigms (LLM-as-a-Judge, reward models, and process reward models)<n>We find that process-level verification does not consistently improve performance and frequently exhibits high variance.
  arXiv  Detail & Related papers  (2026-02-03T03:30:36Z)
• Prompt Optimization via Retrieved Reasoning Assets and Multi-Agent Analysis [5.935239028627343]
  We introduce MA-SAPO, a Multi-Agent framework for Score-Aware Prompt Optimization.<n>Compared to prior methods, MA-SAPO explicitly couples evaluation outcomes with structured reasoning to guide systematic edits.<n>By turning evaluation signals into interpretable reasoning chains, MA-SAPO produces prompt refinements that are more transparent, auditable, and controllable.
  arXiv  Detail & Related papers  (2025-10-18T20:21:09Z)
• Automatic Differentiation of Agent-Based Models [3.0989255691168487]
  Agent-based models (ABMs) simulate complex systems by capturing the bottom-up interactions of individual agents.<n>Many complex systems of interest, such as epidemics or financial markets, involve thousands or even millions of agents.<n>We show that automatic differentiation (AD) techniques can effectively alleviate these computational burdens.
  arXiv  Detail & Related papers  (2025-09-03T13:28:33Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.