Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report

• URL: http://arxiv.org/abs/2608.11965v1
• Date: Wed, 12 Aug 2026 11:53:44 GMT
• Title: Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report
• Abstract summary: Gen AI powered by large language models (LLMs) has transformed the software development process.<n>Recent advances have shifted toward multi-agent systems (MAS) that orchestrate multiple-based agents.<n>This paper provides a comprehensive overview of the existing tools and frameworks for implementing MAS in software engineering.
• Score: 3.5548623483904436
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The proliferation of Generative Artificial Intelligence (Gen AI) powered by large language models (LLMs) has transformed the software development process, introducing new paradigms for code generation, debugging, testing, and maintenance. While early applications focused on leveraging single, independent LLMs to assist developers with isolated tasks, recent advances have shifted toward multi-agent systems (MAS) that orchestrate multiple LLM-based agents working collaboratively toward common objectives. Despite their promising potential, using MAS encompasses a set of challenges for developers who have to carefully select the right technology, devise proper coordination rules, and design specific roles for the involved agents. In this paper, we provide a comprehensive overview of the existing tools and frameworks for implementing MAS in software engineering. First, we conducted a quantitative analysis of the most relevant open source MAS frameworks by evaluating their documentation, features, and capabilities from the developers' perspective. Second, we performed a qualitative evaluation of a subset of the selected frameworks by implementing a common use case: the summarization of README.MD files. The findings show that the selected frameworks provide a good coverage of fundamental components of MAS, though advanced features such as telemetry of agents are still missing. In addition, the empirical evaluation shows that there is no significant difference in terms of ROUGE scores considering the summarization task. Finally, we provide a set of lessons learned and challenges that can help researchers and practitioners to select a suitable MAS framework according to their needs.

Related papers

• The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration [58.61149924275458]
  Tool use enables large language models to access external information, invoke software systems, and act in digital environments beyond what can be solved from model parameters alone.<n>As agent systems evolve, the central problem has shifted from isolated invocation to multi-tool orchestration over long trajectories with intermediate state, execution feedback, changing environments, and practical constraints such as safety, cost, and verifiability.<n>We organize the literature around six core dimensions: inference-time planning and execution, training and trajectory construction, safety and control, efficiency under resource constraints, capability completeness in open environments, and benchmark design and evaluation.
  arXiv  Detail & Related papers  (2026-03-24T07:05:05Z)
• LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities [0.03437656066916039]
  This concept paper systematically reviews the emerging paradigm of LLM-based multi-agent systems.<n>We delve into a wide range of topics such as language model selection, SE evaluation benchmarks, state-of-the-art agentic frameworks and communication protocols.
  arXiv  Detail & Related papers  (2026-01-14T19:28:30Z)
• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System [56.40989626804489]
  This survey provides the first holistic analysis of Large Language Models-powered software engineering.<n>We review over 150 recent papers and propose a taxonomy along two key dimensions: (1) Solutions, categorized into prompt-based, fine-tuning-based, and agent-based paradigms, and (2) Benchmarks, including tasks such as code generation, translation, and repair.
  arXiv  Detail & Related papers  (2025-10-10T06:56:50Z)
• Benchmarking LLM-based Agents for Single-cell Omics Analysis [6.915378212190715]
  AI agents offer a paradigm shift, enabling adaptive planning, executable code generation, traceable decisions, and real-time knowledge fusion.<n>We introduce a novel benchmarking evaluation system to rigorously assess agent capabilities in single-cell omics analysis.
  arXiv  Detail & Related papers  (2025-08-16T04:26:18Z)
• A Survey on Code Generation with LLM-based Agents [61.474191493322415]
  Code generation agents powered by large language models (LLMs) are revolutionizing the software development paradigm.<n>LLMs are characterized by three core features.<n>This paper presents a systematic survey of the field of LLM-based code generation agents.
  arXiv  Detail & Related papers  (2025-07-31T18:17:36Z)
• Deep Research Agents: A Systematic Examination And Roadmap [109.53237992384872]
  Deep Research (DR) agents are designed to tackle complex, multi-turn informational research tasks.<n>In this paper, we conduct a detailed analysis of the foundational technologies and architectural components that constitute DR agents.
  arXiv  Detail & Related papers  (2025-06-22T16:52:48Z)
• GUI Agents with Foundation Models: A Comprehensive Survey [91.97447457550703]
  This survey consolidates recent research on (M)LLM-based GUI agents.<n>We identify key challenges and propose future research directions.<n>We hope this survey will inspire further advancements in the field of (M)LLM-based GUI agents.
  arXiv  Detail & Related papers  (2024-11-07T17:28:10Z)
• LLM-Agent-UMF: LLM-based Agent Unified Modeling Framework for Seamless Integration of Multi Active/Passive Core-Agents [0.0]
  We propose a novel LLM-based Agent Unified Modeling Framework (LLM-Agent-UMF) Our framework distinguishes between the different components of an LLM-based agent, setting LLMs and tools apart from a new element, the core-agent. We evaluate our framework by applying it to thirteen state-of-the-art agents, thereby demonstrating its alignment with their functionalities.
  arXiv  Detail & Related papers  (2024-09-17T17:54:17Z)
• Collaboration Dynamics and Reliability Challenges of Multi-Agent LLM Systems in Finite Element Analysis [3.437656066916039]
  How interagent dynamics influence reasoning quality and verification reliability remains unclear.<n>We study these mechanisms using an AutoGen-based multi-agent framework for linear-elastic Finite Element Analysis (FEA)<n>From 1,120 controlled trials, we find that collaboration effectiveness depends more on functional complementarity than team size.
  arXiv  Detail & Related papers  (2024-08-23T23:11:08Z)
• Large Multimodal Agents: A Survey [78.81459893884737]
  Large language models (LLMs) have achieved superior performance in powering text-based AI agents. There is an emerging research trend focused on extending these LLM-powered AI agents into the multimodal domain. This review aims to provide valuable insights and guidelines for future research in this rapidly evolving field.
  arXiv  Detail & Related papers  (2024-02-23T06:04:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.