CocoaBench: Evaluating Unified Digital Agents in the Wild

• URL: http://arxiv.org/abs/2604.11201v2
• Date: Tue, 14 Apr 2026 08:14:40 GMT
• Title: CocoaBench: Evaluating Unified Digital Agents in the Wild
• Abstract summary: We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks.<n>We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones.
• Score: 98.84044600506597
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks [43.36285223987099]
  We introduce UniClawBench, the first capability-driven benchmark to evaluate proactive agents in dynamic, real-world settings.<n>UniClawBench is built around five executor model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination.<n>Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints.
  arXiv  Detail & Related papers  (2026-07-09T17:59:32Z)
• Code as Agent Harness [107.31925305395957]
  In emerging agentic systems, code is no longer only a target output.<n>Code increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification.<n>This survey provides a unified roadmap toward executable, verifiable, and stateful AI agent systems.
  arXiv  Detail & Related papers  (2026-05-18T17:59:03Z)
• The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration [58.61149924275458]
  Tool use enables large language models to access external information, invoke software systems, and act in digital environments beyond what can be solved from model parameters alone.<n>As agent systems evolve, the central problem has shifted from isolated invocation to multi-tool orchestration over long trajectories with intermediate state, execution feedback, changing environments, and practical constraints such as safety, cost, and verifiability.<n>We organize the literature around six core dimensions: inference-time planning and execution, training and trajectory construction, safety and control, efficiency under resource constraints, capability completeness in open environments, and benchmark design and evaluation.
  arXiv  Detail & Related papers  (2026-03-24T07:05:05Z)
• A Cloud-based Multi-Agentic Workflow for Science [0.12314765641075438]
  Large Language Models (LLMs) become ubiquitous across various scientific domains.<n>Their lack of ability to perform complex tasks like running simulations or to make complex decisions limits their utility.<n>We present a domain-agnostic, model-independent workflow for an agentic framework that can act as a scientific assistant while being run entirely on cloud.
  arXiv  Detail & Related papers  (2026-01-18T22:37:09Z)
• Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases [44.366968508477235]
  We introduce the Confucius Code Agent (CCA), a scalable software engineering agent that can operate at large-scales.<n>CCA is built on top of the Confucius SDK, an agent development platform structured around three complementary perspectives.<n>In addition, we introduce a meta-agent that automates the synthesis, evaluation, and refinement of agent configurations.
  arXiv  Detail & Related papers  (2025-12-11T08:05:58Z)
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering [90.84806758077536]
  We introduce textbfLoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess large language models (LLMs) agents in realistic, long-context software engineering.<n>Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations.<n>Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens.
  arXiv  Detail & Related papers  (2025-11-17T23:57:24Z)
• AgentGit: A Version Control Framework for Reliable and Scalable LLM-Powered Multi-Agent Systems [7.408263799616532]
  We present AgentGit, a framework that brings Git-like rollback and branching to multi-agent systems (MAS)<n>We show that AgentGit significantly reduces redundant, runtime and token usage, and supports parallel exploration across multiple branches.<n>This work offers a practical path to more robust MAS design and enables error recovery, safe exploration, computation, and A/B testing in collaborative AI systems.
  arXiv  Detail & Related papers  (2025-11-01T17:11:31Z)
• AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search [58.98450205734779]
  Large language model (LLM) agents have demonstrated strong capabilities across diverse domains.<n>Existing agent search methods suffer from three major limitations.<n>We introduce a comprehensive framework to address these challenges.
  arXiv  Detail & Related papers  (2025-06-06T12:07:23Z)
• MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents [59.825725526176655]
  Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents.<n>Existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition.<n>We introduce MultiAgentBench, a benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios.
  arXiv  Detail & Related papers  (2025-03-03T05:18:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.