MASEval: Extending Multi-Agent Evaluation from Models to Systems

• URL: http://arxiv.org/abs/2603.08835v1
• Date: Mon, 09 Mar 2026 18:46:17 GMT
• Title: MASEval: Extending Multi-Agent Evaluation from Models to Systems
• Abstract summary: MASEval is a framework-agnostic library that treats the entire system as the unit of analysis.<n>It allows researchers to explore all components of agentic systems, opening new avenues for principled system design.
• Score: 77.09381093359598
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case. MASEval is available under the MIT licence https://github.com/parameterlab/MASEval.

Related papers

• The Illusion of Agentic Complexity in README.md Generation: Evaluating Single-Agent vs. Multi-Agent RAG Systems [3.7149543017244238]
  This study empirically evaluates Retrieval-Augmented Generation (RAG) dependent architectures for the generation of files for GitHub repositories.<n>Results indicate a critical architectural trade-off: the Single-Agent pipeline achieves lexical quality comparable to MAS while reducing token consumption by 86% and operating at twice the speed.
  arXiv  Detail & Related papers  (2026-06-29T16:29:31Z)
• A Unified Framework for the Evaluation of LLM Agentic Capabilities [36.43241368835721]
  We present a unified framework for the fair evaluation of LLM agentic capabilities.<n> Driven by a unified configuration system, the framework integrates diverse benchmarks into a standardized instruction--tool--environment format.<n>We conduct a large-scale empirical analysis over 400K rollouts and 5B tokens on 15 models.
  arXiv  Detail & Related papers  (2026-05-27T03:20:45Z)
• OpenCompass: A Universal Evaluation Platform for Large Language Models [62.59671563145442]
  General-purpose large language models (LLMs) have become a critical link in advancing technological development.<n>Mainstream benchmark datasets face challenges such as the diversity of task types, inconsistent evaluation criteria, and fragmentation of data and processing.<n>This paper proposes and open-source Open LLMs, a one-stop, scalable, and high-concurrency general-purpose evaluation platform.
  arXiv  Detail & Related papers  (2026-05-19T02:50:11Z)
• Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation [50.22481337087162]
  Referring Video Object (RVOS) aims to segment objects in videos based on textual queries.<n>Refer-Agent is a collaborative multi-agent system with alternating reasoning-reflection mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T14:48:12Z)
• MCP-Orchestrated Multi-Agent System for Automated Disinformation Detection [84.75972919995398]
  This paper presents a multi-agent system that uses relation extraction to detect disinformation in news articles.<n>The proposed Agentic AI system combines four agents: (i) a machine learning agent (logistic regression), (ii) a Wikipedia knowledge check agent, and (iv) a web-scraped data analyzer.<n>Results demonstrate that the multi-agent ensemble achieves 95.3% accuracy with an F1 score of 0.964, significantly outperforming individual agents and traditional approaches.
  arXiv  Detail & Related papers  (2025-08-13T19:14:48Z)
• Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systems [69.95482609893236]
  Large Language Model-based Multi-Agent Systems (MASs) have emerged as a powerful paradigm for tackling complex tasks through collaborative intelligence.<n>We call for a paradigm shift toward emphtopology-aware MASs that explicitly model and dynamically optimize the structure of inter-agent interactions.
  arXiv  Detail & Related papers  (2025-05-28T15:20:09Z)
• MooseAgent: A LLM Based Multi-agent Framework for Automating Moose Simulation [1.951051209076837]
  This paper proposes an automated solution framework, MooseAgent, for the multi-physics simulation framework MOOSE.<n>MooseAgent combines large-scale pre-trained language models (LLMs) with a multi-agent system.<n>Results show that MooseAgent can automate the MOOSE simulation process to a certain extent.
  arXiv  Detail & Related papers  (2025-04-11T15:25:50Z)
• Why Do Multi-Agent LLM Systems Fail? [87.90075668488434]
  We introduce MAST-Data, a comprehensive dataset of 1600+ annotated traces collected across 7 popular MAS frameworks.<n>We build the first Multi-Agent System Failure taxonomy (MAST)<n>We leverage MAST and MAST-Data to analyze failure patterns across models (GPT4, Claude 3, Qwen2.5, CodeLlama) and tasks (coding, math, general agent)
  arXiv  Detail & Related papers  (2025-03-17T19:04:38Z)
• The BrowserGym Ecosystem for Web Agent Research [151.90034093362343]
  BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents.<n>We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature.<n>We conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks.
  arXiv  Detail & Related papers  (2024-12-06T23:43:59Z)
• From Models to Systems: A Comprehensive Fairness Framework for Compositional Recommender Systems [6.295527354699332]
  Real-world recommendation systems are built on multiple models and even multiple stages.<n>We propose a holistic framework for modeling system-level fairness.<n>We empirically demonstrate the effectiveness of our proposed framework on synthetic and real datasets.
  arXiv  Detail & Related papers  (2024-12-05T22:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.