A Large-Scale Study on the Development and Issues of Multi-Agent AI Systems

• URL: http://arxiv.org/abs/2601.07136v1
• Date: Mon, 12 Jan 2026 02:07:15 GMT
• Title: A Large-Scale Study on the Development and Issues of Multi-Agent AI Systems
• Abstract summary: This paper analyzes over 42K unique commits and over 4.7K resolved issues across eight leading systems.<n>Our analysis identifies three distinct development profiles: sustained, steady, and burst-driven.<n>Data about issues shows that the most frequent concerns involve bugs (22%), infrastructure (14%), and agent coordination challenges (10%).
• Score: 2.1215369927766714
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid emergence of multi-agent AI systems (MAS), including LangChain, CrewAI, and AutoGen, has shaped how large language model (LLM) applications are developed and orchestrated. However, little is known about how these systems evolve and are maintained in practice. This paper presents the first large-scale empirical study of open-source MAS, analyzing over 42K unique commits and over 4.7K resolved issues across eight leading systems. Our analysis identifies three distinct development profiles: sustained, steady, and burst-driven. These profiles reflect substantial variation in ecosystem maturity. Perfective commits constitute 40.8% of all changes, suggesting that feature enhancement is prioritized over corrective maintenance (27.4%) and adaptive updates (24.3%). Data about issues shows that the most frequent concerns involve bugs (22%), infrastructure (14%), and agent coordination challenges (10%). Issue reporting also increased sharply across all frameworks starting in 2023. Median resolution times range from under one day to about two weeks, with distributions skewed toward fast responses but a minority of issues requiring extended attention. These results highlight both the momentum and the fragility of the current ecosystem, emphasizing the need for improved testing infrastructure, documentation quality, and maintenance practices to ensure long-term reliability and sustainability.

Related papers

• DeepResearch Agent System [0.919770842662809]
  DeepResearch Agent System is a large language model system engineered for deep information retrieval, multi-step reasoning, and autonomous research tasks.<n>System supports a 128K-token context window with hierarchical attention mechanisms that yield 18.7% accuracy and 23.4% recall improvements.<n>System is fully open-sourced, including data synthesis, training, and inference code, and supports applications in academic research, business analysis, R&D support, and education.
  arXiv  Detail & Related papers  (2026-07-30T01:15:47Z)
• Quality and Security Signals in AI-Generated Python Refactoring Pull Requests [0.0]
  We conduct an empirical study of Python pull requests (PRs) from the AIDev dataset.<n>We analyze agentic PRs using PyQu, an ML-based quality assessment tool for Python.<n>Our results show that, on average, agentic commits improve a quality attribute in 22.5% of the studied changes.<n>At the same time, 24.17% of modified files introduce new Pylint issues predominantly convention level violations.
  arXiv  Detail & Related papers  (2026-05-20T17:43:36Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits [3.9532936038777144]
  We present Needle in the Repo (NITR), a framework for evaluating whether behaviorally correct repository edits preserve maintainable structure.<n>NITR distills recurring software engineering wisdom into controlled probes embedded in small, realistic multi-files.<n>We evaluate 23 coding configurations across GPT, Claude, Gemini, and Qwen families in both direct-inference and agent-based settings.
  arXiv  Detail & Related papers  (2026-03-29T15:56:05Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Architecture-Aware Multi-Design Generation for Repository-Level Feature Addition [53.50448142467294]
  RAIM is a multi-design and architecture-aware framework for repository-level feature addition.<n>It shifts away from linear patching by generating multiple diverse implementation designs.<n>Experiments on the NoCode-bench Verified dataset demonstrate that RAIM establishes a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-02T12:50:40Z)
• Data Annotation Quality Problems in AI-Enabled Perception System Development [3.716862357836751]
  Data annotation is essential but highly error-prone in the development of AI-enabled perception systems.<n>We develop a taxonomy of 18 recurring annotation error types across three data-quality dimensions.<n>This study contributes to SE4AI by offering a shared vocabulary, diagnostic toolset, and actionable guidance for building trustworthy AI-enabled perception systems.
  arXiv  Detail & Related papers  (2025-11-20T14:30:51Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving [62.71545696485824]
  We introduce AGENT KB, a universal memory infrastructure enabling seamless experience sharing across heterogeneous agent frameworks without retraining.<n>AGENT KB aggregates trajectories into a structured knowledge base and serves lightweight APIs.<n>We validate AGENT across major frameworks on GAIA, Humanity's Last Exam, GPQA, and SWE-bench.
  arXiv  Detail & Related papers  (2025-07-08T17:59:22Z)
• Retrieval-Augmented Generation with Conflicting Evidence [57.66282463340297]
  Large language model (LLM) agents are increasingly employing retrieval-augmented generation (RAG) to improve the factuality of their responses.<n>In practice, these systems often need to handle ambiguous user queries and potentially conflicting information from multiple sources.<n>We propose RAMDocs (Retrieval with Ambiguity and Misinformation in Documents), a new dataset that simulates complex and realistic scenarios for conflicting evidence for a user query.
  arXiv  Detail & Related papers  (2025-04-17T16:46:11Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• A Comprehensive Study of Bug-Fix Patterns in Autonomous Driving Systems [16.72158049599736]
  We present an empirical study that investigates bug-fix patterns in autonomous driving systems (ADSes)<n>We analyze the commit histories and bug reports of two major autonomous driving projects, Apollo and Autoware, from 1,331 bug fixes.<n>Our study reveals several dominant bug-fix patterns, including those related to path planning, data flow, and configuration management.
  arXiv  Detail & Related papers  (2025-02-04T02:13:05Z)
• Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement [62.94719119451089]
  Lingma SWE-GPT series learns from and simulating real-world code submission activities. Lingma SWE-GPT 72B resolves 30.20% of GitHub issues, marking a significant improvement in automatic issue resolution.
  arXiv  Detail & Related papers  (2024-11-01T14:27:16Z)
• Generative AI for Requirements Engineering: A Systematic Literature Review [1.6986294649170766]
  Generative pretrained transformer models dominate current applications.<n>Industrial adoption remains nascent, with over 90% of studies corresponding to early stage development.<n>Despite the transformative potential of GenAI based RE, several barriers hinder practical adoption.
  arXiv  Detail & Related papers  (2024-09-10T02:44:39Z)
• Cumulative Reasoning with Large Language Models [12.267474250936123]
  Cumulative Reasoning (CR) is a structured framework that enhances large language models (LLMs) problem-solving.<n>CR orchestrates LLMs in three distinct roles--Proposer, Verifier(s), and Reporter--to systematically decompose tasks, generate and validate intermediate reasoning steps, and compose them into a solution.
  arXiv  Detail & Related papers  (2023-08-08T16:18:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.