JobBench: Aligning Agent Work With Human Will

• URL: http://arxiv.org/abs/2605.26329v1
• Date: Mon, 25 May 2026 21:07:02 GMT
• Title: JobBench: Aligning Agent Work With Human Will
• Abstract summary: JobBench evaluates AI agents on the that experts identify as high-priority for delegation.<n>JobBench covers 130 agentic tasks across 35 occupations.
• Score: 121.39878639038716
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus~4.7 under Claude Code, reaches only 45.9 %. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.

Related papers

• Efficient Test-Time Adaptation through Human-AI Interaction [112.5131603022122]
  We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
  arXiv  Detail & Related papers  (2026-09-03T17:33:18Z)
• WorkBench Revisited: Workplace Agents Two Years On [1.2691047660244335]
  The best agent on WorkBench in March 2024, GPT-4, completed 43% of tasks and took an unintended harmful action, such as emailing the wrong person, on 26% of them.<n>We re-visit the benchmark in June 2026 and find that the best agent to date, Claude Opus 4.8, completes 89% and takes an unintended harmful action on 2.5%.
  arXiv  Detail & Related papers  (2026-06-10T21:21:35Z)
• EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents [75.01735520608075]
  Existing benchmarks mostly test literal Theory of Mind (ToM) by asking direct belief questions.<n>We introduce EnactToM, an evolving benchmark of 300 embodied multi-agent tasks set in a 3D household.<n>On the hard split, all seven evaluated frontier models score 0.0% Pass3 on functional task completion, while averaging 45.0% on literal belief probes.
  arXiv  Detail & Related papers  (2026-05-11T00:04:19Z)
• LongSeeker: Elastic Context Orchestration for Long-Horizon Search Agents [63.75550089045995]
  Long-horizon search agents must manage a rapidly growing working context as they reason, call tools, and observe information.<n>We propose that effective context management should be adaptive: parts of the agent's trajectory are maintained at different levels of detail depending on their current relevance to the task.<n>We develop LongSeeker, a long-horizon search agent fine-tuned from Qwen3-30B-A3B on 10k synthesized trajectories.
  arXiv  Detail & Related papers  (2026-05-06T17:54:16Z)
• Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies [29.138677906501865]
  We introduce Workspace-Bench, a benchmark for evaluating AI agents on Workspace Learning involving Large-Scale File Dependencies.<n>We construct realistic workspaces with 5 worker profiles, 74 file types, 20,476 files (up to 20GB) and curate 388 tasks, each with its own file dependency graph, evaluated across 7,399 total rubrics.
  arXiv  Detail & Related papers  (2026-05-05T10:17:06Z)
• EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings [8.341628877927244]
  We introduce EnterpriseOps-Gym, a benchmark designed to evaluate agentic planning in realistic enterprise settings.<n>Within this environment, agents are evaluated on 1,150 expert-curated tasks across eight mission-critical verticals.<n>Our evaluation of 14 frontier models reveals critical limitations in state-of-the-art models.
  arXiv  Detail & Related papers  (2026-03-13T21:09:39Z)
• How Well Does Agent Development Reflect Real-World Work? [89.17217057358285]
  We study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills.<n>We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated.
  arXiv  Detail & Related papers  (2026-03-01T17:55:49Z)
• EnterpriseBench Corecraft: Training Generalizable Agents on High-Fidelity RL Environments [0.10934862523101825]
  We show that training AI agents on high-fidelity reinforcement learning environments produces capabilities that generalize beyond the training distribution.<n>We introduce CoreCraft, the first environment in EnterpriseBench, Surge AI's suite of agentic RL environments.
  arXiv  Detail & Related papers  (2026-02-18T04:35:46Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models [33.1538965735133]
  Cybench is a framework for specifying cybersecurity tasks and evaluating agents on those tasks.<n>We include 40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF competitions.<n>We construct a cybersecurity agent and evaluate 8 models: GPT-4o, OpenAI o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct, Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct.
  arXiv  Detail & Related papers  (2024-08-15T17:23:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.