ClawArena: Benchmarking AI Agents in Evolving Information Environments

• URL: http://arxiv.org/abs/2604.04202v1
• Date: Sun, 05 Apr 2026 17:55:23 GMT
• Title: ClawArena: Benchmarking AI Agents in Evolving Information Environments
• Abstract summary: ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
• Score: 61.664633997138004
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is scattered across heterogeneous sources that often contradict one another, new information can invalidate earlier conclusions, and user preferences surface through corrections rather than explicit instructions. Existing benchmarks largely assume static, single-authority settings and do not evaluate whether agents can keep up with this complexity. We introduce ClawArena, a benchmark for evaluating AI agents in evolving information environments. Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces across multi-channel sessions, workspace files, and staged updates. Evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization, whose interactions yield a 14-category question taxonomy. Two question formats, multi-choice (set-selection) and shell-based executable checks, test both reasoning and workspace grounding. The current release contains 64 scenarios across 8 professional domains, totaling 1{,}879 evaluation rounds and 365 dynamic updates. Experiments on five agent frameworks and five language models show that both model capability (15.4% range) and framework design (9.2%) substantially affect performance, that self-evolving skill frameworks can partially close model-capability gaps, and that belief revision difficulty is determined by update design strategy rather than the mere presence of updates. Code is available at https://github.com/aiming-lab/ClawArena.

Related papers

• A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation [0.9918136378822991]
  We propose a three-tier persona vector with 23 operationalized dimensions.<n>We evaluate the persona model inside a synthetic data generation pipeline across 64,698 conversations.
  arXiv  Detail & Related papers  (2026-09-08T11:28:02Z)
• KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents [5.481534695653804]
  We introduce KC-Bench, a controlled multi-turn benchmark for measuring this capability.<n>Its 238 tasks are manually screened from more than 1,000 generated candidates.<n>No model handles factual correction, identity consistency checking, and temporal conflict resolution reliably across all settings.
  arXiv  Detail & Related papers  (2026-09-03T09:35:14Z)
• A Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent Adoption [33.31789388282026]
  Coding agents increase development velocity but also technical debt.<n>Prior work reports only average effects across adopters, hiding wide differences between teams.<n>We introduce RAMP, a four-level cumulative maturity model grounded in version-controlled artifacts that teams commit to configure AI tools.
  arXiv  Detail & Related papers  (2026-08-26T00:00:48Z)
• Kozuchi Agent: A Language-Agnostic Open-Weight Agent for Software Repair [13.668549452625117]
  Kozuchi Agent is a language-agnostic open-weight repair agent and CI-operated evaluation pipeline.<n>Explicit phases, persistent state, deterministic tools, a model-independent action interface, and cross-agent test-time selection make runs auditable and repeatable.
  arXiv  Detail & Related papers  (2026-08-16T07:06:37Z)
• AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research [39.68061830142623]
  AutoWorldModel-Bench is a benchmark for AI coding agents acting as autonomous researchers.<n>It spans eight game environments under a unified structured-state representation--ground-truth entity state extracted from each game.<n>Our benchmark offers a setting in which frontier coding agents can be evaluated on open-ended research rather than engineering-to-spec problems.
  arXiv  Detail & Related papers  (2026-07-20T23:35:45Z)
• How to Interpret Agent Behavior [56.59836196946289]
  We introduce ACT*ONOMY, a taxonomy for describing and analyzing agent behavior at runtime.<n>By providing a shared vocabulary, ACT*ONOMY helps researchers, agent designers, and end users interpret agent behavior more consistently.
  arXiv  Detail & Related papers  (2026-05-13T14:52:40Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Re-Evaluating EVMBench: Are AI Agents Ready for Smart Contract Security? [10.248746359119625]
  EVMbench is the first large-scale benchmark for AI agents on smart contract security.<n>Its results have fueled expectations that fully automated AI auditing is within reach.<n>These findings challenge the narrative that fully automated AI auditing is imminent.
  arXiv  Detail & Related papers  (2026-03-11T14:07:16Z)
• BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? [61.247730037229815]
  We introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope.<n>To investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities.<n>This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
  arXiv  Detail & Related papers  (2026-03-03T17:52:01Z)
• ESAA: Event Sourcing for Autonomous Agents in LLM-Based Software Engineering [0.0]
  This paper presents the ESAA (Event Sourcing for Autonomous Agents) architecture.<n>The architecture separates the cognitive intention from the project's state mutation, inspired by the Event Sourcing pattern.<n>Two case studies validate the architecture, providing empirical evidence of the architecture's scalability beyond single-agent scenarios.
  arXiv  Detail & Related papers  (2026-02-26T16:45:59Z)
• PublicAgent: Multi-Agent Design Principles From an LLM-Based Open Data Analysis Framework [5.863391019411233]
  Large language models show promise for individual tasks, but end-to-end analytical expose fundamental limitations.<n>We present PublicAgent, a multi-agent framework that addresses these limitations through decomposition into specialized agents for intent clarification, dataset discovery, analysis, and reporting.
  arXiv  Detail & Related papers  (2025-11-04T21:48:11Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.