MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?

• URL: http://arxiv.org/abs/2606.23664v1
• Date: Mon, 22 Jun 2026 17:48:40 GMT
• Title: MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?
• Abstract summary: Multi-agent systems (MAS) offer a scalable path forward for agentic AI.<n>System prompts specify agents' roles and behaviors, enabling system-level improvements without model finetuning.<n>It remains unclear whether, when, and by how much prompt optimization improves MAS performance.
• Score: 12.31270556465256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-agent systems (MAS) offer a scalable path forward for agentic AI, comprising multiple LLM-based agents, each assigned a system prompt and a position within a workflow that governs inter-agent coordination and output aggregation. System prompts thus form a critical and accessible optimization surface: they specify agents' roles and behaviors, enabling system-level improvements without model finetuning. Although prompt optimization has shown substantial potential for single LLMs, extending it to MAS poses distinct challenges, notably an exponentially growing search space. It remains unclear whether, when, and by how much prompt optimization improves MAS performance, and how sensitive such gains are to system configuration. In this work, we systematically study system-prompt optimization across a broad range of MAS setups varying in task, workflow, communication protocol, and team size, benchmarking two prompt optimizers that naturally extend state-of-the-art single-agent methods. The results reveal its potential to unlock significant gains while exposing open challenges, characterizing when and how much prompt optimization helps across diverse MAS settings.

Related papers

• Agent-GWO: Collaborative Agents for Dynamic Prompt Optimization in Large Language Models [69.55139736609367]
  Agent-GWO is a dynamic prompt optimization framework for complex reasoning.<n>We show that Agent-GWO consistently improves accuracy and stability over existing prompt optimization methods.
  arXiv  Detail & Related papers  (2026-04-14T07:35:37Z)
• MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inference [40.87762929194057]
  Multi-Agent PRompt Optimization (MAPRO) is a framework that first formulates MAS prompt optimization as a Maximum a Posteriori (MAP) inference problem.<n>To address credit assignment and updates the system iteratively, MAPRO employs a topology-aware refinement mechanism.<n>Across benchmarks in various tasks, MAPRO achieves state-of-the-art performance, consistently surpassing manually engineered baselines.
  arXiv  Detail & Related papers  (2025-10-08T19:21:24Z)
• Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development [14.245155142689988]
  We perform an empirical case study on group optimization of role-based multi-agent systems utilizing natural language feedback.<n>We propose a two-step agent prompts optimization pipeline: identifying underperforming agents and optimizing system prompts of identified agents.<n>Overall, we demonstrate the effectiveness of our optimization method for role-based multi-agent systems tackling software development tasks evaluated on diverse evaluation dimensions.
  arXiv  Detail & Related papers  (2025-05-22T00:00:27Z)
• System Prompt Optimization with Meta-Learning [64.74765550805024]
  We introduce the novel problem of bilevel system prompt optimization, whose objective is to design system prompts that are robust to diverse user prompts.<n>We propose a meta-learning framework, which meta-learns the system prompt by optimizing it over various user prompts across multiple datasets.<n>We conduct experiments on 14 unseen datasets spanning 5 different domains, on which we show that our approach produces system prompts that generalize effectively to diverse user prompts.
  arXiv  Detail & Related papers  (2025-05-14T16:46:15Z)
• Towards more Contextual Agents: An extractor-Generator Optimization Framework [0.0]
  Large Language Model (LLM)-based agents have demonstrated remarkable success in solving complex tasks across a wide range of general-purpose applications.<n>However, their performance often degrades in context-specific scenarios, such as specialized industries or research domains.<n>To address this challenge, our work introduces a systematic approach to enhance the contextual adaptability of LLM-based agents.
  arXiv  Detail & Related papers  (2025-02-18T15:07:06Z)
• Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies [41.21314691388456]
  Large language models, employed as multiple agents that interact and collaborate with each other, have excelled at solving complex tasks.<n> Designing prompts and topologies for multi-agent systems (MAS) is inherently complex.<n>We propose Multi-Agent System Search (MASS), a MAS optimization framework that efficiently exploits the complex MAS design space.
  arXiv  Detail & Related papers  (2025-02-04T17:56:44Z)
• SPRIG: Improving Large Language Model Performance by System Prompt Optimization [45.96513122345295]
  Large Language Models (LLMs) have shown impressive capabilities in many scenarios, but their performance depends on the choice of prompt. We propose SPRIG, an edit-based genetic algorithm that iteratively constructs prompts from prespecified components to maximize the model's performance in general scenarios. We evaluate the performance of system prompts on a collection of 47 different types of tasks to ensure generalizability.
  arXiv  Detail & Related papers  (2024-10-18T18:51:44Z)
• Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System [75.25394449773052]
  Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving.<n>Yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods.<n>We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness.
  arXiv  Detail & Related papers  (2024-10-10T17:00:06Z)
• QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning [58.767866109043055]
  We introduce Query-dependent Prompt Optimization (QPO), which iteratively fine-tune a small pretrained language model to generate optimal prompts tailored to the input queries.<n>We derive insights from offline prompting demonstration data, which already exists in large quantities as a by-product of benchmarking diverse prompts on open-sourced tasks.<n> Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency of our method in both zero-shot and few-shot scenarios.
  arXiv  Detail & Related papers  (2024-08-20T03:06:48Z)
• MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization [73.7779735046424]
  We show that different prompts should be adapted to different Large Language Models (LLM) to enhance their capabilities across various downstream tasks in NLP. We then propose a model-adaptive prompt (MAPO) method that optimize the original prompts for each specific LLM in downstream tasks.
  arXiv  Detail & Related papers  (2024-07-04T18:39:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.