ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge

• URL: http://arxiv.org/abs/2609.12366v1
• Date: Fri, 11 Sep 2026 02:36:21 GMT
• Title: ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge
• Abstract summary: ORQA is a method for testing occupation-level knowledge in large language models.<n>It connects O*NET occupations to trusted occupation-specific websites.<n>A combination of an automated pipeline and human review produces a set of high quality questions.
• Score: 5.218089081443592
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present ORQA, a method for testing occupation-level knowledge in large language models. Prior methods either map abstract LLM skills to occupations via task definitions or utilize expert knowledge which is difficult to obtain at scale and expensive. ORQA complements both of these methods by connecting O*NET occupations to trusted occupation-specific websites (such as regulatory agencies, licensing bodies, professional organizations, and government publications) and converting these into source-traceable question-answer pairs. A combination of an automated pipeline and human review produces a set of high quality questions about occupations. The question set created via our method covers 116 occupations from all 21 major groups in the SOC, with 480 questions sourced from 187 different websites. Each question is designed to probe a real-world skill question that is relevant to the occupation in question. We test 15 state-of-the-art frontier and open-weight models via this method. Claude Opus 4.6, GPT-5.4 and Claude Sonnet 4.6 all perform the best at approximately 58-62% while smaller open-weight models achieve approximately 33-41% performance. Performance varies significantly across occupations. Healthcare-related occupations achieve the highest performance (78%) while Office and Administrative Support achieve approximately 40%. Performance on individual occupations (e.g. Sheet Metal Workers and Fish and Game Wardens) is essentially zero. We also find that open-ended questions and weighting by wage bill do not significantly affect the ranking of models on this benchmark. We believe that leveraging existing trusted occupation-specific information to test LLM knowledge in professional domains may be a scalable and useful method for evaluating occupation-level AI performance in the future. Results and data are available at orqabench.org.

Related papers

• Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills [86.15447717549154]
  We present DisCo, a skill-powered research agent that creates skills and uses them during research.<n>We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work.
  arXiv  Detail & Related papers  (2026-09-02T15:49:41Z)
• RubberDuckBench: A Benchmark for AI Coding Assistants [5.198865387380684]
  We present RubberDuckBench: a benchmark of questions about code, along with detailed rubrics for evaluating answers.<n>We evaluate a diverse set of 20 LLMs (proprietary & open-source) on answering these questions.<n>Grok 4 (69.29%), Claude Opus 4 (68.5%), and GPT-5 (67.8%) perform best overall, but do not exhibit pairwise significant superiority over the next 9 best performing models.
  arXiv  Detail & Related papers  (2026-01-23T05:28:48Z)
• Expert Evaluation of LLM World Models: A High-$T_c$ Superconductivity Case Study [8.574376236521958]
  Large Language Models (LLMs) show great promise as a powerful tool for scientific literature exploration.<n>We evaluate the ability of LLM systems to understand the literature at the level of an expert.<n>We construct an expert-curated database of 1,726 scientific papers that covers the history of the field, and a set of 67 expert-formulated questions.
  arXiv  Detail & Related papers  (2025-11-05T19:00:01Z)
• SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection [81.78173888579941]
  Large Language Models (LLMs) are considered a well-suited method to increase the quality of the question-answering functionality.<n>LLMs are trained on web data, where researchers have no control over whether the benchmark or the knowledge graph was already included in the training data.<n>This paper introduces a novel method that evaluates the quality of LLMs by generating a SPARQL query from a natural-language question.
  arXiv  Detail & Related papers  (2025-07-18T12:28:08Z)
• CLR-Bench: Evaluating Large Language Models in College-level Reasoning [17.081788240112417]
  Large language models (LLMs) have demonstrated their remarkable performance across various language understanding tasks. We present CLR-Bench to comprehensively evaluate the LLMs in complex college-level reasoning.
  arXiv  Detail & Related papers  (2024-10-23T04:55:08Z)
• OccuQuest: Mitigating Occupational Bias for Inclusive Large Language Models [73.49209444768057]
  We create an instruction-tuning dataset named emphOccuQuest, which contains 110,000+ prompt-completion pairs and 30,000+ dialogues covering over 1,000 occupations in 26 occupational categories. We then fine-tune LLaMA on OccuQuest to obtain OccuLLaMA, which significantly outperforms state-of-the-art LLaMA variants on professional questions in GPT-4 and human evaluations.
  arXiv  Detail & Related papers  (2023-10-25T10:06:17Z)
• OpsEval: A Comprehensive IT Operations Benchmark Suite for Large Language Models [26.590755599827993]
  We present OpsEval, a comprehensive task-oriented Ops benchmark designed for large language models (LLMs)<n>The benchmark includes 7184 multi-choice questions and 1736 question-answering (QA) formats in English and Chinese.<n>To ensure the credibility of our evaluation, we invite dozens of domain experts to manually review our questions.
  arXiv  Detail & Related papers  (2023-10-11T16:33:29Z)
• Self-Knowledge Guided Retrieval Augmentation for Large Language Models [59.771098292611846]
  Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. Self-Knowledge guided Retrieval augmentation (SKR) is a simple yet effective method which can let LLMs refer to the questions they have previously encountered.
  arXiv  Detail & Related papers  (2023-10-08T04:22:33Z)
• ExpertQA: Expert-Curated Questions and Attributed Answers [51.68314045809179]
  We conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality. We collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.
  arXiv  Detail & Related papers  (2023-09-14T16:54:34Z)
• Do Large Language Models Know What They Don't Know? [74.65014158544011]
  Large language models (LLMs) have a wealth of knowledge that allows them to excel in various Natural Language Processing (NLP) tasks. Despite their vast knowledge, LLMs are still limited by the amount of information they can accommodate and comprehend. This study aims to evaluate LLMs' self-knowledge by assessing their ability to identify unanswerable or unknowable questions.
  arXiv  Detail & Related papers  (2023-05-29T15:30:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.