Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies

• URL: http://arxiv.org/abs/2605.03596v4
• Date: Thu, 14 May 2026 13:14:02 GMT
• Title: Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies
• Abstract summary: We introduce Workspace-Bench, a benchmark for evaluating AI agents on Workspace Learning involving Large-Scale File Dependencies.<n>We construct realistic workspaces with 5 worker profiles, 74 file types, 20,476 files (up to 20GB) and curate 388 tasks, each with its own file dependency graph, evaluated across 7,399 total rubrics.
• Score: 29.138677906501865
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Workspace learning requires AI agents to identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a worker's workspace, enabling them to complete both routine and advanced tasks effectively. Despite its importance, existing relevant benchmarks largely evaluate agents on pre-specified or synthesized files with limited real-world dependencies, leaving workspace-level evaluation underexplored. To this end, we introduce Workspace-Bench, a benchmark for evaluating AI agents on Workspace Learning involving Large-Scale File Dependencies. We construct realistic workspaces with 5 worker profiles, 74 file types, 20,476 files (up to 20GB) and curate 388 tasks, each with its own file dependency graph, evaluated across 7,399 total rubrics that require cross-file retrieval, contextual reasoning, and adaptive decision-making. We further provide Workspace-Bench-Lite, a 100-task subset that preserves the benchmark distribution while reducing evaluation costs by about 70%. We evaluate 4 popular agent harnesses and 7 foundation models. Experimental results show that current agents remain far from reliable workspace learning, where the best reaches only about 60%, substantially below the human result of 80.7%, and the average performance across agents is only 43.3%.

Related papers

• Efficient Test-Time Adaptation through Human-AI Interaction [112.5131603022122]
  We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
  arXiv  Detail & Related papers  (2026-09-03T17:33:18Z)
• Reliable LLM-Generated Programs for High-Energy Physics Experiments through Graph-Grounded Software Knowledge [18.431539281285225]
  General-purpose large language models (LLMs) often produce unreliable programs for such tasks.<n>We organize these software relations before generation and retrieve task-relevant knowledge at inference time.<n>We evaluate a complete grounding system that combines hybrid retrieval over a heterogeneous software knowledge graph.
  arXiv  Detail & Related papers  (2026-09-01T11:35:02Z)
• PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems [53.65117408512214]
  We introduce textbfPersonaForge, a user simulation framework for synthesizing realistic multi-turn user--agent interactions.<n>Using PersonaForge, we construct a 6.3K-record training dataset and textbfPersonaForge-Bench, a manually annotated 138-task benchmark.<n>Experiments show that PersonaForge training improves the composite score by +4.1%, with gains across all four dimensions.
  arXiv  Detail & Related papers  (2026-08-28T14:33:19Z)
• ContextWeave: A Real-World Workflow Benchmark [44.23367918659128]
  ContextWeave is a longitudinal benchmark that evaluates whether recalled experience improves downstream agent performance in realistic office-work streams.<n>It measures quality and alignment with participant-specific preferences, complemented by diagnostics of relevance, continuity, solvability, and robustness to misleading recall.<n>Our analysis shows that actionable, experience-rich memory supports workflow continuation and reduces redundant exploration more effectively than compact summaries.
  arXiv  Detail & Related papers  (2026-08-05T13:31:18Z)
• DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces [25.28248447900504]
  Data agents enable natural-language analytics over organizational workspaces.<n>Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis.<n>We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results.
  arXiv  Detail & Related papers  (2026-08-04T10:48:11Z)
• Learning Compositional Meta-Routing for Agentic Workflows: An Executable Benchmark [0.0]
  We introduce an executable benchmark and a budget-aware meta-router that composes heterogeneous operations from raw task text.<n>Results: learned success falls to 75.9% and trails static routing at 93.5%, while remaining 49% cheaper and exceeding one-shot routing by 34.3 points.
  arXiv  Detail & Related papers  (2026-07-31T07:00:49Z)
• OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks [98.07146747892239]
  OSWorld 2.0 is a benchmark of 108 long-horizon computer-use across everyday and professional tasks.<n>Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete.
  arXiv  Detail & Related papers  (2026-06-28T17:59:17Z)
• Persistent AI Agents in Academic Research: A Single-Investigator Implementation Case Study [0.0]
  The unit of analysis was the persistent human-agent environment: researcher, agent, memory layer, tools, repositories, scheduled jobs, specialized agent roles, and governance rules.<n>A strict May 2026 trajectory subset captured 627 model-related events and 73.95 million recorded tokens, of which 82.9% were cache reads.
  arXiv  Detail & Related papers  (2026-05-26T11:28:36Z)
• Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows [67.92316850084575]
  We introduce Claw-Eval-Live, a live benchmark for workflow agents.<n>Each release is constructed from public workflow-demand signals.<n>Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts.
  arXiv  Detail & Related papers  (2026-04-30T17:23:19Z)
• HippoCamp: Benchmarking Contextual Agents on Personal Computers [71.97629614361549]
  HippoCamp is a new benchmark designed to evaluate agents' capabilities on multimodal file management.<n>Our benchmark instantiates device-scale file systems over real-world profiles spanning diverse modalities, comprising 42.4 GB of data across over 2K real-world files.
  arXiv  Detail & Related papers  (2026-04-01T17:58:33Z)
• EnterpriseBench Corecraft: Training Generalizable Agents on High-Fidelity RL Environments [0.10934862523101825]
  We show that training AI agents on high-fidelity reinforcement learning environments produces capabilities that generalize beyond the training distribution.<n>We introduce CoreCraft, the first environment in EnterpriseBench, Surge AI's suite of agentic RL environments.
  arXiv  Detail & Related papers  (2026-02-18T04:35:46Z)
• ARIAL: An Agentic Framework for Document VQA with Precise Answer Localization [1.580774794371876]
  We present ARIAL, a framework that orchestrates specialized tools to achieve both precise answer extraction and reliable spatial grounding.<n>We evaluate ARIAL on four benchmarks (DocVQA, FUNSD, CORD, and SROIE) using both textual accuracy (ANLS) and spatial precision.<n>Our work demonstrates how agentic orchestration of specialized tools can simultaneously improve performance and interpretability.
  arXiv  Detail & Related papers  (2025-11-22T21:09:28Z)
• Agent READMEs: An Empirical Study of Context Files for Agentic Coding [8.019313057979522]
  We study 2,303 agent context files from 1,925 repositories to characterize their structure, maintenance, and content.<n>We find that these files are not static documentation but complex, difficult-to-read artifacts that evolve like configuration code, maintained through frequent, small additions.<n>These findings indicate that while developers use context files to make agents functional, they provide few guardrails to ensure that agent-written code is secure or performant, highlighting the need for improved tooling and practices.
  arXiv  Detail & Related papers  (2025-11-17T02:18:55Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments [2.184775414778289]
  We introduce setupbench, a benchmark that isolates the environment-bootstrap skill.<n>Our tasks span seven language ecosystems, five database engines, and multi-service orchestration scenarios.<n>We find low success rates across task categories, with particular challenges in repository setup (38.9-57.4%) and local database configuration (20.0-53.3%)
  arXiv  Detail & Related papers  (2025-07-11T22:45:07Z)
• FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks [52.47895046206854]
  FieldWorkArena is a benchmark for agentic AI targeting real-world field work.<n>This paper defines a new action space that agentic AI should possess for real world work environment benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T08:21:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.