xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations

• URL: http://arxiv.org/abs/2506.13651v1
• Date: Mon, 16 Jun 2025 16:16:14 GMT
• Title: xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations
• Abstract summary: We introduce xbench, a profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity.<n>xbench targets commercially significant domains with evaluation tasks defined by industry professionals.<n>We present initial evaluation results for leading contemporary agents, establishing a baseline for these professional domains.
• Score: 30.34618272227522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce xbench, a dynamic, profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity. While existing benchmarks often focus on isolated technical skills, they may not accurately reflect the economic value agents deliver in professional settings. To address this, xbench targets commercially significant domains with evaluation tasks defined by industry professionals. Our framework creates metrics that strongly correlate with productivity value, enables prediction of Technology-Market Fit (TMF), and facilitates tracking of product capabilities over time. As our initial implementations, we present two benchmarks: Recruitment and Marketing. For Recruitment, we collect 50 tasks from real-world headhunting business scenarios to evaluate agents' abilities in company mapping, information retrieval, and talent sourcing. For Marketing, we assess agents' ability to match influencers with advertiser needs, evaluating their performance across 50 advertiser requirements using a curated pool of 836 candidate influencers. We present initial evaluation results for leading contemporary agents, establishing a baseline for these professional domains. Our continuously updated evalsets and evaluations are available at https://xbench.org.

Related papers

• A Framework for Evaluating Agentic Skills at Scale [0.4509560087514856]
  Agent skills are structured, reusable knowledge artifacts that augment agent capabilities.<n>No reusable methodology exists for evaluating an individual skill.<n>We present an evaluation framework that lets a skill author construct realistic tasks.
  arXiv  Detail & Related papers  (2026-06-16T11:46:56Z)
• AlphaEval: Evaluating Agents in Production [32.52315519888507]
  We present AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies deploying AI agents in their core business.<n>Unlike model-centric benchmarks, AlphaEval evaluates complete agent products as commercial systems, capturing performance variations invisible to model-level evaluation.
  arXiv  Detail & Related papers  (2026-04-14T00:43:20Z)
• How Well Does Agent Development Reflect Real-World Work? [89.17217057358285]
  We study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills.<n>We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated.
  arXiv  Detail & Related papers  (2026-03-01T17:55:49Z)
• AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents [11.506625665006979]
  AD-Bench is a benchmark based on real-world business requirements of advertising and marketing platforms.<n>The benchmark categorizes requests into three difficulty levels to evaluate agents' capabilities under multi-round, multi-tool collaboration.<n>Experiments show that even state-of-the-art models still exhibit substantial capability gaps in complex advertising and marketing analysis scenarios.
  arXiv  Detail & Related papers  (2026-02-15T17:59:47Z)
• Benchmarking Agents in Insurance Underwriting Environments [0.9728664856449597]
  Existing benchmarks overemphasize open-domains such as code, use narrow accuracy metrics, and lack authentic complexity.<n>We present UNDERWRITE, an expert-first, multi-turn insurance underwriting benchmark designed in close collaboration with domain experts.
  arXiv  Detail & Related papers  (2026-01-31T02:12:11Z)
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering [90.84806758077536]
  We introduce textbfLoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess large language models (LLMs) agents in realistic, long-context software engineering.<n>Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations.<n>Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens.
  arXiv  Detail & Related papers  (2025-11-17T23:57:24Z)
• UpBench: A Dynamically Evolving Real-World Labor-Market Agentic Benchmark Framework Built for Human-Centric AI [2.0619484032730813]
  UpBench is a benchmark grounded in real jobs drawn from the global Upwork labor marketplace.<n>Each task corresponds to a verified client transaction, anchoring evaluation in genuine work activity and financial outcomes.<n>UpBench employs a rubric-based evaluation framework, in which expert freelancers decompose each job into detailed, verifiable acceptance criteria and assess AI submissions with per-criterion feedback.
  arXiv  Detail & Related papers  (2025-11-15T17:39:37Z)
• Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents [1.0305173936249623]
  This white paper proposes a novel framework of eleven outcome-based, task-agnostic performance metrics for AI agents.<n>We introduce metrics such as Goal Completion Rate (GCR), Autonomy Index (AIx), Multi-Step Task Resilience (MTR), and Business Impact Efficiency (BIE)<n>Our results reveal significant performance trade-offs between different agent designs, highlighting the Hybrid Agent as the most consistently high-performing model.
  arXiv  Detail & Related papers  (2025-11-11T13:40:46Z)
• FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks [52.47895046206854]
  FieldWorkArena is a benchmark for agentic AI targeting real-world field work.<n>This paper defines a new action space that agentic AI should possess for real world work environment benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T08:21:46Z)
• CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions [85.88573535033406]
  CRMArena-Pro is a novel benchmark for holistic, realistic assessment of LLM agents in diverse professional settings.<n>It incorporates multi-turn interactions guided by diverse personas and robust confidentiality awareness assessments.<n>Experiments reveal leading LLM agents achieve only around 58% single-turn success on CRMArena-Pro, with performance dropping significantly to approximately 35% in multi-turn settings.
  arXiv  Detail & Related papers  (2025-05-24T21:33:22Z)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites [9.58858258192147]
  We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites.<n>We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions.<n>Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation.
  arXiv  Detail & Related papers  (2025-04-15T18:22:55Z)
• AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories [59.214178488091584]
  We propose AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents.<n>Using our benchmark, we evaluate 12 LLM judges and find that no single LLM excels across all benchmarks.<n>We also find that the rule-based evaluation used by common benchmarks tends to underreport the success rate of web agents.
  arXiv  Detail & Related papers  (2025-04-11T19:49:22Z)
• An Illusion of Progress? Assessing the Current State of Web Agents [49.76769323750729]
  We conduct a comprehensive and rigorous assessment of the current state of web agents.<n>Results depict a very different picture of the competency of current agents, suggesting over-optimism in previously reported results.<n>We introduce Online-Mind2Web, an online evaluation benchmark consisting of 300 diverse and realistic tasks spanning 136 websites.
  arXiv  Detail & Related papers  (2025-04-02T05:51:29Z)
• Grounded Persuasive Language Generation for Automated Marketing [59.175257431078435]
  This paper develops an agentic framework that employs large language models (LLMs) to automate the generation of persuasive and grounded marketing content.<n>Our method is designed to align the generated content with user preferences while highlighting useful factual attributes.<n>We conduct systematic human-subject experiments in the domain of real estate marketing, with a focus group of potential house buyers.
  arXiv  Detail & Related papers  (2025-02-24T03:36:57Z)
• More than Marketing? On the Information Value of AI Benchmarks for Practitioners [42.73526862595375]
  In academia, public benchmarks were generally viewed as suitable measures for capturing research progress.<n>In product and policy, benchmarks were often found to be inadequate for informing substantive decisions.<n>We conclude that effective benchmarks should provide meaningful, real-world evaluations, incorporate domain expertise, and maintain transparency in scope and goals.
  arXiv  Detail & Related papers  (2024-12-07T03:35:39Z)
• CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments [90.29937153770835]
  We introduce CRMArena, a benchmark designed to evaluate AI agents on realistic tasks grounded in professional work environments.<n>We show that state-of-the-art LLM agents succeed in less than 40% of the tasks with ReAct prompting, and less than 55% even with function-calling abilities.<n>Our findings highlight the need for enhanced agent capabilities in function-calling and rule-following to be deployed in real-world work environments.
  arXiv  Detail & Related papers  (2024-11-04T17:30:51Z)
• AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising [19.642481233488667]
  We propose AdTEC (Ad Text Evaluation Benchmark by CyberAgent), the first public benchmark to evaluate ad texts from multiple perspectives.<n>Our contributions are as follows: (i) Defining five tasks for evaluating the quality of ad texts, as well as building a Japanese dataset based on the practical operational experiences of advertising agencies, which are typically kept in-house.
  arXiv  Detail & Related papers  (2024-08-12T03:32:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.