WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain

• URL: http://arxiv.org/abs/2604.13055v1
• Date: Tue, 17 Mar 2026 15:56:42 GMT
• Title: WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain
• Abstract summary: We present textbfWorkRB (Work Research Benchmark), the first open-source,stakeholder-driven benchmark tailored to work-domain AI.<n>WorkRB organizes 13 diverse tasks from 7 task groups as unified recommendation and NLP tasks, including job/skill recommendation, candidate recommendation, similar item recommendation, and skill extraction and normalization.<n>WorkRB has a modular design for seamless contributions and integration of proprietary tasks without disclosing sensitive data.
• Score: 37.1490186890843
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Today's evolving labor markets rely increasingly on recommender systems for hiring, talent management, and workforce analytics, with natural language processing (NLP) capabilities at the core. Yet, research in this area remains highly fragmented. Studies employ divergent ontologies (ESCO, O*NET, national taxonomies), heterogeneous task formulations, and diverse model families, making cross-study comparison and reproducibility exceedingly difficult. General-purpose benchmarks lack coverage of work-specific tasks, and the inherent sensitivity of employment data further limits open evaluation. We present \textbf{WorkRB} (Work Research Benchmark), the first open-source, community-driven benchmark tailored to work-domain AI. WorkRB organizes 13 diverse tasks from 7 task groups as unified recommendation and NLP tasks, including job/skill recommendation, candidate recommendation, similar item recommendation, and skill extraction and normalization. WorkRB enables both monolingual and cross-lingual evaluation settings through dynamic loading of multilingual ontologies. Developed within a multi-stakeholder ecosystem of academia, industry, and public institutions, WorkRB has a modular design for seamless contributions and enables integration of proprietary tasks without disclosing sensitive data. WorkRB is available under the Apache 2.0 license at https://github.com/techwolf-ai/WorkRB.

Related papers

• TalentCLEF at CLEF2026: Skill and Job Title Intelligence for Human Capital Management [0.20209739009228475]
  This paper presents the second edition of the TalentCLEF Challenge, which will run as an evaluation lab as part of CLEF 2026.<n>The aim of TalentCLEF is to promote the development of systems and methods that use Natural Language Processing (NLP) in the field of Human Capital Management (HCM)<n>To this end, TalentCLEF establishes benchmarks where research teams can compare methods and share findings.
  arXiv  Detail & Related papers  (2026-07-22T10:51:28Z)
• Unified Work Embeddings: Contrastive Learning of a Bidirectional Multi-task Ranker [3.4204762278595346]
  We introduce WorkBench, the first unified evaluation suite spanning six work-related tasks formulated explicitly as ranking problems.<n>We use this insight to compose task-specific bipartite graphs from real-world data, synthetically enriched through grounding.<n>This leads to Unified Work Embeddings (UWE), a task-agnostic bi-encoder that exploits our training-data structure with a many-to-many InfoNCE objective.
  arXiv  Detail & Related papers  (2025-11-11T08:28:26Z)
• SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models [59.90381306452982]
  evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer.<n>We introduce SWE-1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework.<n>SWE- spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests.
  arXiv  Detail & Related papers  (2025-11-07T18:01:32Z)
• MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks [56.34018316319873]
  We propose MERA Code, a benchmark for evaluating code for the latest code generation LLMs in Russian.<n>This benchmark includes 11 evaluation tasks that span 8 programming languages.<n>We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages.
  arXiv  Detail & Related papers  (2025-07-16T14:31:33Z)
• FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks [52.47895046206854]
  FieldWorkArena is a benchmark for agentic AI targeting real-world field work.<n>This paper defines a new action space that agentic AI should possess for real world work environment benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T08:21:46Z)
• R-Eval: A Unified Toolkit for Evaluating Domain Knowledge of Retrieval Augmented Large Language Models [51.468732121824125]
  Large language models have achieved remarkable success on general NLP tasks, but they may fall short for domain-specific problems. Existing evaluation tools only provide a few baselines and evaluate them on various domains without mining the depth of domain knowledge. In this paper, we address the challenges of evaluating RALLMs by introducing the R-Eval toolkit, a Python toolkit designed to streamline the evaluation of different RAGs.
  arXiv  Detail & Related papers  (2024-06-17T15:59:49Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain [26.045871822474723]
  This study introduces a language model called ESCOXLM-R, based on XLM-R, which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations taxonomy. We evaluate the performance of ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets.
  arXiv  Detail & Related papers  (2023-05-20T04:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.