Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation

• URL: http://arxiv.org/abs/2604.02368v1
• Date: Fri, 27 Mar 2026 11:28:15 GMT
• Title: Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation
• Abstract summary: Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks.<n>Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases.<n>We present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains.
• Score: 32.5154721488471
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators.

Related papers

• EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks [0.0]
  We dedicate more than 4,000 human expert hours to evaluate a selection of six frontier LLMs on a member of this class of problems.<n>We see that the best way to extract this kind of conclusion is by employing human evaluators, carefully checking their consistency through rigorous statistical analysis, and observe that automatic measurements also fall short when evaluating on this case of real-world open-ended problems with a subjective ground truth.
  arXiv  Detail & Related papers  (2026-08-05T07:39:49Z)
• Traditional statistical representations outperform generative AI in identifying expert peer reviewers [22.555352928759206]
  The exponential growth of scientific submissions has strained the peer review system.<n>We evaluate the reliability of statistical and AI-driven expertise identification methodologies.<n>Our results highlight that distinguishing subfield expertise requires fine-grained vocabulary.
  arXiv  Detail & Related papers  (2026-05-18T17:59:45Z)
• An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems [1.9138416746729587]
  Large Language Models (LLMs) are transforming scholarly tasks like search and summarization, but their reliability remains uncertain.<n>We developed a schema for evaluating LLM errors in question-answering systems that reflects the assessment strategies of practicing scientists.
  arXiv  Detail & Related papers  (2026-02-24T16:16:44Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• DEER: A Comprehensive and Reliable Benchmark for Deep-Research Expert Reports [49.217247659479476]
  deep research systems can generate expert-level reports via multi-step reasoning and evidence-based synthesis.<n>Existing benchmarks often lack systematic criteria for expert reporting.<n>We introduce DEER, a benchmark for evaluating expert-level deep research reports.
  arXiv  Detail & Related papers  (2025-12-19T16:46:20Z)
• PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning [18.32501228579171]
  Professional Reasoning Bench (PRBench) is a realistic, open-ended, and difficult benchmark of real-world problems in Finance and Law.<n>We open-source its 1,100 expert-authored tasks and 19,356 expert-curated criteria, making it the largest public, rubric-based benchmark for both legal and finance domains.
  arXiv  Detail & Related papers  (2025-11-14T18:55:12Z)
• ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge [94.40918390309186]
  evaluating progress in large language models (LLMs) is often constrained by the challenge of verifying responses.<n>We introduce ProfBench: a set of over 7000 response-criterion pairs as evaluated by human-experts.<n>Our findings reveal that ProfBench poses significant challenges even for state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-10-21T17:59:44Z)
• Curse of Knowledge: When Complex Evaluation Context Benefits yet Biases LLM Judges [72.3356133063925]
  The paradigm of large language models (LLMs) as judges has emerged as a scalable solution, yet prior work primarily focuses on simple settings.<n>Our in-depth analysis offers crucial insights for improving the accuracy and verifiability of evaluation signals.
  arXiv  Detail & Related papers  (2025-09-03T15:48:33Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines [118.8024915014751]
  Large language models (LLMs) have demonstrated remarkable proficiency in academic disciplines such as mathematics, physics, and computer science.<n>However, human knowledge encompasses over 200 specialized disciplines, far exceeding the scope of existing benchmarks.<n>We present SuperGPQA, a benchmark that evaluates graduate-level knowledge and reasoning capabilities across 285 disciplines.
  arXiv  Detail & Related papers  (2025-02-20T17:05:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.