Economic Evaluations of Language Models

• URL: http://arxiv.org/abs/2607.19375v1
• Date: Fri, 26 Jun 2026 22:26:40 GMT
• Title: Economic Evaluations of Language Models
• Abstract summary: EconEvals is an open-source evaluation suite to measure capabilities relevant to tasks, work activities, and occupations in the US labor economy.<n>Our evaluations improve coverage over OpenAI's GDPval benchmark, which is the existing state-of-the-art that covers 5% of US occupations, at 500x lower cost.<n>Our estimates indicate that current models could save workers substantial time on at least half of their tasks in 47% of occupations.
• Score: 81.69183693497281
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models perform economically valuable work, yet they are not currently assessed for how well they perform every economically valuable task. We introduce EconEvals as an open-source evaluation suite to measure capabilities relevant to tasks, work activities, and occupations in the US labor economy. We ground the evaluation suite in real user queries to language models where possible, and supplement these with synthetic data. Our evaluations improve coverage over OpenAI's GDPval benchmark, which is the existing state-of-the-art that covers 5% of US occupations, at 500x lower cost. Alongside benchmarks, we also introduce a simulation-based exposure measure to estimate how much time current language model capabilities could save across all tasks belonging to all US occupations, with detailed accounting for each estimate. Our estimates indicate that current models could save workers substantial time on at least half of their tasks in 47% of occupations. However, for 79% of tasks where we predict substantial time savings, observed Claude usage is low, suggesting that existing usage lags potential. Beyond inherent constraints of language model chatbots, our data identifies privacy and proprietary systems as the principal bottlenecks limiting further time savings from AI. Overall, we introduce adaptable infrastructure that grounds inferences about language models' labor-market impact in their current capabilities, which can be continually updated as capabilities improve.

Related papers

• WorldBench: Culturally Grounded Benchmark for Multilingual Agents [69.76002914143531]
  Existing benchmarks rarely test state preservation, performance across languages, and application to realistic, grounded scenarios.<n>We present WorldBench: a comprehensive, multilingual benchmark, where agents can act in a sandbox via structured actions.<n>WorldBench comprises 1,600 tasks across seven languages and eight cultures, filtered and refined through feedback from human annotators.
  arXiv  Detail & Related papers  (2026-09-01T10:53:07Z)
• The Open Source Economic Index of AI Adoption and Capability [27.900759713490576]
  We work towards measuring both AI adoption and the capability of AI to perform discrete labor tasks across various occupations.<n>We develop an open-source economic index that uses publicly available user-LLM chat data and O*NET tasks to replicate studies produced by frontier AI labs.<n>We build a system that generates benchmark scenarios grounded in O*NET occupations, tasks, and model-context-protocol (MCP) servers.
  arXiv  Detail & Related papers  (2026-05-23T18:45:42Z)
• Automated Analysis of Sustainability Reports: Using Large Language Models for the Extraction and Prediction of EU Taxonomy-Compliant KPIs [21.656551146954587]
  Large Language Models (LLMs) offer a path to automation.<n>We introduce a novel, structured dataset from 190 corporate reports.<n>Our results reveal a clear performance gap between qualitative and quantitative tasks.
  arXiv  Detail & Related papers  (2025-12-30T15:28:03Z)
• How AI Forecasts AI Jobs: Benchmarking LLM Predictions of Labor Market Changes [5.848712585343904]
  This paper introduces a benchmark for evaluating how well large language models (LLMs) can anticipate changes in job demand.<n>Our benchmark combines two datasets: a high-frequency index of sector-level job postings in the United States, and a global dataset of projected occupational changes due to AI adoption.<n>Results show that structured task prompts consistently improve forecast stability, while persona prompts offer advantages on short-term trends.
  arXiv  Detail & Related papers  (2025-10-27T14:08:27Z)
• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks [34.63352957472507]
  GDPval is a benchmark evaluating AI capabilities on real-world economically valuable tasks.<n>GDPval covers the majority of U.S. Bureau of Labor Statistics Work Activities for 44 occupations across the top 9 sectors contributing to U.S. GDP.<n>We analyze the potential for frontier models, when paired with human oversight, to perform GDPval tasks cheaper and faster than unaided experts.
  arXiv  Detail & Related papers  (2025-10-05T21:36:43Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context.<n>We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters.<n>We propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content [62.816876067499415]
  We propose LiveXiv: a scalable evolving live benchmark based on scientific ArXiv papers.<n>LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate visual question-answer pairs.<n>We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models true abilities.
  arXiv  Detail & Related papers  (2024-10-14T17:51:23Z)
• LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages [1.149936119867417]
  Low-resource languages face significant barriers in AI development due to limited linguistic resources and expertise for data labeling. We propose leveraging the potential of LLMs in the active learning loop for data annotation. Empirical evaluations, notably employing GPT-4-Turbo, demonstrate near-state-of-the-art performance with significantly reduced data requirements.
  arXiv  Detail & Related papers  (2024-04-02T19:34:22Z)
• Privacy Adhering Machine Un-learning in NLP [66.17039929803933]
  In real world industry use Machine Learning to build models on user data. Such mandates require effort both in terms of data as well as model retraining. continuous removal of data and model retraining steps do not scale. We propose textitMachine Unlearning to tackle this challenge.
  arXiv  Detail & Related papers  (2022-12-19T16:06:45Z)
• Forecasting Future World Events with Neural Networks [68.43460909545063]
  Autocast is a dataset containing thousands of forecasting questions and an accompanying news corpus. The news corpus is organized by date, allowing us to precisely simulate the conditions under which humans made past forecasts. We test language models on our forecasting task and find that performance is far below a human expert baseline.
  arXiv  Detail & Related papers  (2022-06-30T17:59:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.