LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review

• URL: http://arxiv.org/abs/2604.16321v1
• Date: Wed, 25 Feb 2026 07:55:49 GMT
• Title: LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review
• Abstract summary: Large Language Models (LLMs) have enabled multi-agent systems to perform autonomous code generation for complex tasks.<n>Despite the recent growth in research and industrial applications in this area, there is little work on synthesizing evidence from both academic and industrial sources.
• Score: 0.7257685311746803
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have enabled multi-agent systems to perform autonomous code generation for complex tasks. Despite the recent growth in research and industrial applications in this area, there is little work on synthesizing evidence from both academic and industrial sources to capture the current state of research on LLM-based multi-agent systems for code generation. To this end, we conducted a Multi-Vocal Literature Review (MLR), combining insights from both academia and industry, including peer-reviewed studies and grey literature. The aim of this study is to systematically synthesize and analyze existing knowledge on LLM-based multi-agent systems for code generation. Specifically, the review examines the motivations for their use, employed benchmarks and models, key challenges, proposed solutions, and potential directions for future research. We selected and reviewed 114 studies, and the key findings are: 1) the identified reasons for adopting multi-agent systems for code generation were classified into nine categories; 2) the models and evaluation benchmarks utilized across the studies were systematically analyzed to provide a structured overview of commonly adopted LLM configurations and assessment practices; 3) the reported challenges and corresponding solutions were synthesized into six main categories and 26 subcategories; and 4) future research directions were identified and organized into six main categories and 18 subcategories. The results of this MLR will assist researchers and practitioners in pursuing further studies and supporting the real-world adoption of multi-agent systems in industrial settings.

Related papers

• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System [56.40989626804489]
  This survey provides the first holistic analysis of Large Language Models-powered software engineering.<n>We review over 150 recent papers and propose a taxonomy along two key dimensions: (1) Solutions, categorized into prompt-based, fine-tuning-based, and agent-based paradigms, and (2) Benchmarks, including tasks such as code generation, translation, and repair.
  arXiv  Detail & Related papers  (2025-10-10T06:56:50Z)
• LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios [63.08653028889316]
  We propose a systematic taxonomy that decomposes agentic reasoning frameworks and analyze how these frameworks dominate framework-level reasoning.<n>Specifically, we propose an unified formal language to further classify agentic reasoning systems into single-agent methods, tool-based methods, and multi-agent methods.<n>We provide a comprehensive review of their key application scenarios in scientific discovery, healthcare, software engineering, social simulation, and economics.
  arXiv  Detail & Related papers  (2025-08-25T06:01:16Z)
• A Survey on Code Generation with LLM-based Agents [61.474191493322415]
  Code generation agents powered by large language models (LLMs) are revolutionizing the software development paradigm.<n>LLMs are characterized by three core features.<n>This paper presents a systematic survey of the field of LLM-based code generation agents.
  arXiv  Detail & Related papers  (2025-07-31T18:17:36Z)
• Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey [69.45421620616486]
  This work presents the first structured taxonomy and analysis of discrete tokenization methods designed for large language models (LLMs)<n>We categorize 8 representative VQ variants that span classical and modern paradigms and analyze their algorithmic principles, training dynamics, and integration challenges with LLM pipelines.<n>We identify key challenges including codebook collapse, unstable gradient estimation, and modality-specific encoding constraints.
  arXiv  Detail & Related papers  (2025-07-21T10:52:14Z)
• LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research [32.35279830326718]
  Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery.<n>However, their capability in reproducing code from research papers, especially in the NLP domain, remains underexplored.<n>We present LMR-BENCH, a benchmark designed to evaluate the capability of LLM agents on code reproduction from Language Modeling Research.
  arXiv  Detail & Related papers  (2025-06-19T07:04:16Z)
• From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems [6.284317913684068]
  Compound Al Systems (CAIS) is an emerging paradigm that integrates large language models (LLMs) with external components, such as retrievers, agents, tools, and orchestrators.<n>Despite growing adoption in both academia and industry, the CAIS landscape remains fragmented, lacking a unified framework for analysis, taxonomy, and evaluation.<n>This survey aims to provide researchers and practitioners with a comprehensive foundation for understanding, developing, and advancing the next generation of system-level artificial intelligence.
  arXiv  Detail & Related papers  (2025-06-05T02:34:43Z)
• ResearchCodeAgent: An LLM Multi-Agent System for Automated Codification of Research Methodologies [16.90884865239373]
  We introduce ResearchCodeAgent, a novel multi-agent system to automate the codification of research methodologies.<n>The system bridges the gap between high-level research concepts and their practical implementation.<n>ResearchCodeAgent represents a significant step towards the research implementation process, potentially accelerating the pace of machine learning research.
  arXiv  Detail & Related papers  (2025-04-28T07:18:45Z)
• IdeaBench: Benchmarking Large Language Models for Research Idea Generation [19.66218274796796]
  Large Language Models (LLMs) have transformed how people interact with artificial intelligence (AI) systems. We propose IdeaBench, a benchmark system that includes a comprehensive dataset and an evaluation framework. Our dataset comprises titles and abstracts from a diverse range of influential papers, along with their referenced works. Our evaluation framework is a two-stage process: first, using GPT-4o to rank ideas based on user-specified quality indicators such as novelty and feasibility, enabling scalable personalization.
  arXiv  Detail & Related papers  (2024-10-31T17:04:59Z)
• Retrieval-Enhanced Machine Learning: Synthesis and Opportunities [60.34182805429511]
  Retrieval-enhancement can be extended to a broader spectrum of machine learning (ML) This work introduces a formal framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by synthesizing the literature in various domains in ML with consistent notations which is missing from the current literature. The goal of this work is to equip researchers across various disciplines with a comprehensive, formally structured framework of retrieval-enhanced models, thereby fostering interdisciplinary future research.
  arXiv  Detail & Related papers  (2024-07-17T20:01:21Z)
• A Software Engineering Perspective on Testing Large Language Models: Research, Practice, Tools and Benchmarks [2.8061460833143346]
  Large Language Models (LLMs) are rapidly becoming ubiquitous both as stand-alone tools and as components of current and future software systems. To enable usage of LLMs in the high-stake or safety-critical systems of 2030, they need to undergo rigorous testing.
  arXiv  Detail & Related papers  (2024-06-12T13:45:45Z)
• Large Multimodal Agents: A Survey [78.81459893884737]
  Large language models (LLMs) have achieved superior performance in powering text-based AI agents. There is an emerging research trend focused on extending these LLM-powered AI agents into the multimodal domain. This review aims to provide valuable insights and guidelines for future research in this rapidly evolving field.
  arXiv  Detail & Related papers  (2024-02-23T06:04:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.