Design and Report Benchmarks for Knowledge Work

• URL: http://arxiv.org/abs/2605.23262v1
• Date: Fri, 22 May 2026 06:03:01 GMT
• Title: Design and Report Benchmarks for Knowledge Work
• Abstract summary: This paper contributes a three-step approach for making explicit how benchmarked tasks represent the work claims attached to their scores.<n>To name the work activity being evaluated and distinguish it from common benchmark tasks, we derive an inventory of 18 work activities from the O*NET occupational task database.
• Score: 5.13982016225783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The development of LLM agents has led to a growing body of work on knowledge-work AI, including coding, research, and healthcare. However, current knowledge-work evaluation and benchmark design still largely follow the logic of traditional NLP tasks. As a result, higher benchmark performance does not reliably show that a system can carry out knowledge work in real-world deployment settings. This paper contributes a three-step approach for making explicit how benchmarked tasks represent the work claims attached to their scores: defining the work activity under evaluation, specifying the tested setting, and scoring the appropriate work product. We review work studies showing that knowledge work is organized through roles and responsibilities, local materials and tools, and artifacts that must remain usable in downstream workflows. We then translate these concerns into benchmark design and reporting guidance, covering how tasks should be mapped to work activities, how tested settings should specify materials, tools, roles, and constraints, and how scoring should focus on the work product left by the system. To name the work activity being evaluated and distinguish it from common benchmark tasks, we derive an inventory of 18 work activities from the O{*}NET occupational task database. We demonstrate the approach through three benchmark case analyses: GDPval, a non-code occupational deliverable benchmark; OfficeQA Pro, a grounded document-analysis benchmark scored by final answers; and APEX-SWE, a software-engineering benchmark with executable scored products. These cases show how benchmark design choices shape the strongest work claim a score can support, and where gaps arise between the benchmarked task, tested setting, scored product, and broader work claim.

Related papers

• DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents [9.592889418944608]
  We present DI-Bench, a pipeline for generating realistic benchmarks for data intelligence (DI)<n>The pipeline produces a 731-task benchmark covering knowledge retrieval, analytical computation, and rule-grounded reasoning.<n>We evaluate four models, revealing a substantial finding: models achieve only 32% accuracy when doing computational tasks where retrieved business rules modify the computation.
  arXiv  Detail & Related papers  (2026-09-04T23:53:21Z)
• From Question Answering to Task Completion: A Survey on Agent System and Harness Design [67.8241530947325]
  This survey examines LLM-based agents through a model-harness lens.<n>We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model and an execution harness.<n>We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities.
  arXiv  Detail & Related papers  (2026-06-14T05:40:16Z)
• ProdCodeBench: A Production-Derived Benchmark for Evaluating AI Coding Agents [10.578603956693696]
  This paper presents a methodology for curating production-derived benchmarks, illustrated through ProdCodeBench.<n>Each curated sample consists of a verbatim prompt, a committed code change and fail-to-pass tests spanning seven programming languages.<n>Our systematic analysis of four foundation models yields solve rates ranging from 53.2% to 72.2%.
  arXiv  Detail & Related papers  (2026-04-02T01:52:55Z)
• Characterizing Knowledge Graph Tasks in LLM Benchmarks Using Cognitive Complexity Frameworks [0.13108652488669734]
  Large Language Models (LLMs) are increasingly used for tasks involving Knowledge Graphs (KGs)<n>We propose a complementary task characterization approach using three complexity frameworks from cognitive psychology.<n>Applying this to the LLM-KG-Bench framework, we highlight value distributions, identify underrepresented demands and motivate richer interpretation and diversity for benchmark evaluation tasks.
  arXiv  Detail & Related papers  (2025-09-17T08:15:14Z)
• DOCUEVAL: An LLM-based AI Engineering Tool for Building Customisable Document Evaluation Workflows [14.403156040714817]
  We present DOCUEVAL, an AI engineering tool for building customisable evaluators.<n>We show how DOCUEVAL enables both the engineering of evaluators and scalable, reliable document evaluation.
  arXiv  Detail & Related papers  (2025-09-12T08:09:09Z)
• Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
  We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
  arXiv  Detail & Related papers  (2025-08-05T12:52:09Z)
• Establishing Best Practices for Building Rigorous Agentic Benchmarks [94.69724201080155]
  We show that many agentic benchmarks have issues in task setup or reward design.<n>Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms.<n>We introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience.
  arXiv  Detail & Related papers  (2025-07-03T17:35:31Z)
• FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks [52.47895046206854]
  FieldWorkArena is a benchmark for agentic AI targeting real-world field work.<n>This paper defines a new action space that agentic AI should possess for real world work environment benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T08:21:46Z)
• A Task-Centric Perspective on Recommendation Systems [32.44458308850838]
  We analyze RecSys task formulations, emphasizing key components such as input-output structures, temporal dynamics, and candidate item selection.<n>We explore the balance between task specificity and model generalizability, highlighting how well-defined task formulations serve as the foundation for robust evaluation and effective solution development.
  arXiv  Detail & Related papers  (2025-03-27T06:10:22Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models [7.3673614578648285]
  Large Language Models (LLMs) excel in code-related tasks like code generation, but benchmark evaluations often overlook task characteristics, such as difficulty.<n>This paper introduces a framework using diverse prompts and Item Response Theory (IRT) to efficiently assess LLMs' capabilities and benchmark task characteristics.
  arXiv  Detail & Related papers  (2024-07-30T22:31:19Z)
• ECBD: Evidence-Centered Benchmark Design for NLP [95.50252564938417]
  We propose Evidence-Centered Benchmark Design (ECBD), a framework which formalizes the benchmark design process into five modules. Each module requires benchmark designers to describe, justify, and support benchmark design choices. Our analysis reveals common trends in benchmark design and documentation that could threaten the validity of benchmarks' measurements.
  arXiv  Detail & Related papers  (2024-06-13T00:59:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.