OAgents: An Empirical Study of Building Effective Agents

• URL: http://arxiv.org/abs/2506.15741v2
• Date: Mon, 23 Jun 2025 09:22:39 GMT
• Title: OAgents: An Empirical Study of Building Effective Agents
• Authors: He Zhu, Tianrui Qin, King Zhu, Heyuan Huang, Yeyi Guan, Jinxiang Xia, Yi Yao, Hanhao Li, Ningning Wang, Pai Liu, Tianhao Peng, Xin Gui, Xiaowan Li, Yuhui Liu, Yuchen Eleanor Jiang, Jun Wang, Changwang Zhang, Xiangru Tang, Ge Zhang, Jian Yang, Minghao Liu, Xitong Gao, Jiaheng Liu, Wangchunshu Zhou,
• Abstract summary: We study the impact of popular design choices in key agent components in a fair and rigorous manner.<n>Based on our findings, we build and open-source OAgents, a new foundation agent framework.
• Score: 46.50371876218872
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recently, Agentic AI has become an increasingly popular research field. However, we argue that current agent research practices lack standardization and scientific rigor, making it hard to conduct fair comparisons among methods. As a result, it is still unclear how different design choices in agent frameworks affect effectiveness, and measuring their progress remains challenging. In this work, we conduct a systematic empirical study on GAIA benchmark and BrowseComp to examine the impact of popular design choices in key agent components in a fair and rigorous manner. We find that the lack of a standard evaluation protocol makes previous works, even open-sourced ones, non-reproducible, with significant variance between random runs. Therefore, we introduce a more robust evaluation protocol to stabilize comparisons. Our study reveals which components and designs are crucial for effective agents, while others are redundant, despite seeming logical. Based on our findings, we build and open-source OAgents, a new foundation agent framework that achieves state-of-the-art performance among open-source projects. OAgents offers a modular design for various agent components, promoting future research in Agentic AI.

Related papers

• AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent [57.10083973844841]
  AgentArk is a novel framework to distill multi-agent dynamics into the weights of a single model.<n>We investigate three hierarchical distillation strategies across various models, tasks, scaling, and scenarios.<n>By shifting the burden of computation from inference to training, the distilled models preserve the efficiency of one agent while exhibiting strong reasoning and self-correction performance of multiple agents.
  arXiv  Detail & Related papers  (2026-02-03T19:18:28Z)
• An Empirical Study of Agent Developer Practices in AI Agent Frameworks [59.862193600499914]
  The rise of large language models (LLMs) has sparked a surge of interest in agents, leading to the rapid growth of agent frameworks.<n>Despite widespread use of agent frameworks, their practical applications and how they influence the agent development process remain underexplored.<n>More than 80% of developers report difficulties in identifying the frameworks that best meet their specific development requirements.
  arXiv  Detail & Related papers  (2025-12-01T17:52:15Z)
• What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity [40.27555449103923]
  We examine the role that ideation diversity plays in agent performance.<n>Different models and agent scaffolds yield varying degrees of ideation diversity.<n> higher-performing agents tend to have increased ideation diversity.
  arXiv  Detail & Related papers  (2025-11-19T16:32:18Z)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite [75.58737079136942]
  We present AstaBench, a suite that provides the first holistic measure of agentic ability to perform scientific research.<n>Our suite comes with the first scientific research environment with production-grade search tools.<n>Our evaluation of 57 agents across 22 agent classes reveals several interesting findings.
  arXiv  Detail & Related papers  (2025-10-24T17:10:26Z)
• How can we assess human-agent interactions? Case studies in software agent design [52.953425368394306]
  We make two major steps towards the rigorous assessment of human-agent interactions.<n>We propose PULSE, a framework for more efficient human-centric evaluation of agent designs.<n>We deploy the framework on a large-scale web platform built around the open-source software agent OpenHands.
  arXiv  Detail & Related papers  (2025-10-10T19:04:28Z)
• Inefficiencies of Meta Agents for Agent Design [25.46718879564119]
  We examine three key challenges in a common class of meta-agents.<n>First, we investigate how a meta-agent learns across iterations.<n>Second, although the meta-agent designs multiple agents during training, it typically commits to a single agent at test time.
  arXiv  Detail & Related papers  (2025-10-08T07:06:17Z)
• An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications [12.166151903597445]
  Foundation model (FM)-based AI agents are rapidly gaining adoption across diverse domains.<n>Their inherent non-determinism and non-reproducibility pose testing and quality assurance challenges.<n>We conduct the first large-scale empirical study of testing practices in the AI agent ecosystem.
  arXiv  Detail & Related papers  (2025-09-23T16:02:09Z)
• ConfAgents: A Conformal-Guided Multi-Agent Framework for Cost-Efficient Medical Diagnosis [11.18347744454527]
  We introduce HealthFlow, a self-evolving AI agent that overcomes limitations through a novel meta-level evolution mechanism.<n>HealthFlow autonomously refines its own high-level problem-solving policies by distilling procedural successes and failures into a durable, strategic knowledge base.<n>Our experiments demonstrate that HealthFlow's self-evolving approach significantly outperforms state-of-the-art agent frameworks.
  arXiv  Detail & Related papers  (2025-08-06T22:39:38Z)
• Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training [67.895981259683]
  General AI Agents are increasingly recognized as foundational frameworks for the next generation of artificial intelligence.<n>Current agent systems are either closed-source or heavily reliant on a variety of paid APIs and proprietary tools.<n>We present Cognitive Kernel-Pro, a fully open-source and (to the maximum extent) free multi-module agent framework.
  arXiv  Detail & Related papers  (2025-08-01T08:11:31Z)
• Deep Research Agents: A Systematic Examination And Roadmap [79.04813794804377]
  Deep Research (DR) agents are designed to tackle complex, multi-turn informational research tasks.<n>In this paper, we conduct a detailed analysis of the foundational technologies and architectural components that constitute DR agents.
  arXiv  Detail & Related papers  (2025-06-22T16:52:48Z)
• Understanding Software Engineering Agents Through the Lens of Traceability: An Empirical Study [15.97770416681533]
  Software engineering agents (SWE agents) operate autonomously by interpreting user input and responding to environmental feedback.<n>We present the first systematic study of SWE agent behavior through the lens of execution traces.
  arXiv  Detail & Related papers  (2025-06-10T00:41:54Z)
• Unifying Language Agent Algorithms with Graph-based Orchestration Engine for Reproducible Agent Research [32.92036657863354]
  Language agents powered by large language models (LLMs) have demonstrated remarkable capabilities in understanding, reasoning, and executing complex tasks.<n>However, developing robust agents presents significant challenges: substantial engineering overhead, lack of standardized components, and insufficient evaluation frameworks for fair comparison.<n>We introduce Agent Graph-based Orchestration for Reasoning and Assessment (AGORA), a flexible and abstraction framework that addresses these challenges.
  arXiv  Detail & Related papers  (2025-05-30T08:46:23Z)
• Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness [50.29739337771454]
  Multi-agent debate (MAD) approaches offer improved reasoning, robustness, and diverse perspectives over monolithic models.<n>This paper conceptualizes MAD as a test-time computational scaling technique, distinguished by collaborative refinement and diverse exploration capabilities.<n>We conduct a comprehensive empirical investigation comparing MAD with strong self-agent test-time scaling baselines on mathematical reasoning and safety-related tasks.
  arXiv  Detail & Related papers  (2025-05-29T01:02:55Z)
• Large Language Model Agent: A Survey on Methodology, Applications and Challenges [88.3032929492409]
  Large Language Model (LLM) agents, with goal-driven behaviors and dynamic adaptation capabilities, potentially represent a critical pathway toward artificial general intelligence.<n>This survey systematically deconstructs LLM agent systems through a methodology-centered taxonomy.<n>Our work provides a unified architectural perspective, examining how agents are constructed, how they collaborate, and how they evolve over time.
  arXiv  Detail & Related papers  (2025-03-27T12:50:17Z)
• Enhancing Heterogeneous Multi-Agent Cooperation in Decentralized MARL via GNN-driven Intrinsic Rewards [1.179778723980276]
  Multi-agent Reinforcement Learning (MARL) is emerging as a key framework for sequential decision-making and control tasks. The deployment of these systems in real-world scenarios often requires decentralized training, a diverse set of agents, and learning from infrequent environmental reward signals. We propose the CoHet algorithm, which utilizes a novel Graph Neural Network (GNN) based intrinsic motivation to facilitate the learning of heterogeneous agent policies.
  arXiv  Detail & Related papers  (2024-08-12T21:38:40Z)
• Fact-based Agent modeling for Multi-Agent Reinforcement Learning [6.431977627644292]
  Fact-based Agent modeling (FAM) method is proposed in which fact-based belief inference (FBI) network models other agents in partially observable environment only based on its local information. We evaluate FAM on various Multiagent Particle Environment (MPE) and compare the results with several state-of-the-art MARL algorithms.
  arXiv  Detail & Related papers  (2023-10-18T19:43:38Z)
• ProAgent: Building Proactive Cooperative Agents with Large Language Models [89.53040828210945]
  ProAgent is a novel framework that harnesses large language models to create proactive agents. ProAgent can analyze the present state, and infer the intentions of teammates from observations. ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various coordination scenarios.
  arXiv  Detail & Related papers  (2023-08-22T10:36:56Z)
• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents [2.5204420653245245]
  Multi-Agent Reinforcement Learning (MARL) has been widely applied in many fields such as smart traffic and unmanned aerial vehicles. This work proposes a novel Robustness Testing framework for MARL that attacks states of Critical Agents.
  arXiv  Detail & Related papers  (2023-06-09T02:26:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.