\$OneMillion-Bench: How Far are Language Agents from Human Experts?

• URL: http://arxiv.org/abs/2603.07980v1
• Date: Mon, 09 Mar 2026 05:32:42 GMT
• Title: \$OneMillion-Bench: How Far are Language Agents from Human Experts?
• Abstract summary: $OneMillion-Bench is a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science.<n>$OneMillion-Bench provides a unified testbed for assessing agentic reliability, professional depth, and practical readiness in domain-intensive scenarios.
• Score: 43.31541643833527
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce \$OneMillion-Bench \$OneMillion-Bench, a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focused on expert-level problems to ensure meaningful differentiation across agents. Together, \$OneMillion-Bench provides a unified testbed for assessing agentic reliability, professional depth, and practical readiness in domain-intensive scenarios.

Related papers

• INS-ActBench: A Comprehensive Benchmark for Assessing Professional Actuarial Capability of Large Language Models [11.556217835954842]
  INS-ActBench is a benchmark for evaluating actuarial capability in large language models (LLMs)<n> INS-ActBench contains 12,050 Q&A pairs from public exams and sample questions released by 16 actuarial associations.<n>It covers three subsets: textbfINS-Act-Know for standardized actuarial knowledge, textbfINS-Act-Case for long-context insurance case reasoning, and textbfINS-Act-Practice for spreadsheet and R-code tasks with numerical outputs.
  arXiv  Detail & Related papers  (2026-07-27T11:12:50Z)
• Herculean: An Agentic Benchmark for Financial Intelligence [114.33879935277217]
  We introduce Herculean, the first skilled benchmark for agentic financial intelligence spanning four representative skill sets.<n>Across frontier agents, we find agents perform relatively well on Trading and Market Insights, but struggle substantially on Hedging and Auditing.<n>Overall, our results point to a key gap in current agents in turning financial reasoning into dependable workflow execution.
  arXiv  Detail & Related papers  (2026-05-14T04:30:49Z)
• FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks [4.444378412245679]
  We introduce FrontierFinance, a benchmark of 25 complex financial modeling tasks across five core finance models.<n>We engage human experts to define the tasks, create rubrics, grade LLMs, and perform the tasks themselves as human baselines.<n>We demonstrate that our human experts both receive higher scores on average, and are more likely to provide client-ready outputs than current state-of-the-art systems.
  arXiv  Detail & Related papers  (2026-04-07T14:15:45Z)
• $τ$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge [58.03692489021332]
  $$-Knowledge is an extension of $$-Bench for evaluating agents in environments where success depends on coordinating external, natural-language knowledge with tool outputs.<n>We show that $$-Knowledge provides a realistic testbed for developing agents that integrate unstructured knowledge in human-facing deployments.
  arXiv  Detail & Related papers  (2026-03-04T18:34:47Z)
• MeetBench-XL: Calibrated Multi-Dimensional Evaluation and Learned Dual-Policy Agents for Real-Time Meetings [17.012108640696916]
  We introduce MeetAll, a bilingual and multimodal corpus derived from enterprise meetings totaling 140 hours.<n>MeetBench XL measures factual fidelity, intent alignment, response efficiency, structural clarity, and completeness.<n>MeetMaster XL is a learned dual policy agent that jointly optimize query routing between fast and slow reasoning paths and tool invocation.
  arXiv  Detail & Related papers  (2026-02-03T09:08:18Z)
• Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systems [54.916243942641444]
  Large language models (LLMs) are emerging as key enablers of automation in domains such as telecommunications.<n>We study an edge-cloud-expert cascaded LLM-based knowledge system that supports decision-making through a question-and-answer pipeline.
  arXiv  Detail & Related papers  (2025-12-23T03:10:09Z)
• Benchmarking LLM Agents for Wealth-Management Workflows [0.0]
  This dissertation extends TheAgentCompany with a finance-focused environment.<n>It investigates whether a general purpose LLM agent can complete representative wealth-management tasks both accurately and economically.
  arXiv  Detail & Related papers  (2025-12-01T21:56:21Z)
• UpBench: A Dynamically Evolving Real-World Labor-Market Agentic Benchmark Framework Built for Human-Centric AI [2.0619484032730813]
  UpBench is a benchmark grounded in real jobs drawn from the global Upwork labor marketplace.<n>Each task corresponds to a verified client transaction, anchoring evaluation in genuine work activity and financial outcomes.<n>UpBench employs a rubric-based evaluation framework, in which expert freelancers decompose each job into detailed, verifiable acceptance criteria and assess AI submissions with per-criterion feedback.
  arXiv  Detail & Related papers  (2025-11-15T17:39:37Z)
• PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning [18.32501228579171]
  Professional Reasoning Bench (PRBench) is a realistic, open-ended, and difficult benchmark of real-world problems in Finance and Law.<n>We open-source its 1,100 expert-authored tasks and 19,356 expert-curated criteria, making it the largest public, rubric-based benchmark for both legal and finance domains.
  arXiv  Detail & Related papers  (2025-11-14T18:55:12Z)
• How Good are Foundation Models in Step-by-Step Embodied Reasoning? [79.15268080287505]
  Embodied agents must make decisions that are safe, spatially coherent, and grounded in context.<n>Recent advances in large multimodal models have shown promising capabilities in visual understanding and language generation.<n>Our benchmark includes over 1.1k samples with detailed step-by-step reasoning across 10 tasks and 8 embodiments.
  arXiv  Detail & Related papers  (2025-09-18T17:56:30Z)
• FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
  FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
  arXiv  Detail & Related papers  (2025-08-07T22:15:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.