From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

• URL: http://arxiv.org/abs/2608.09925v1
• Date: Mon, 10 Aug 2026 17:59:05 GMT
• Title: From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
• Abstract summary: "Grip on LLMs" is a systematic evaluation suite for Dutch governmental use.<n>We identify six evaluation dimensions (factuality, honesty, social bias, energy consumption, cost, and training data transparency)<n>We operationalise them into a benchmark suite covering more than 30 multilingual and Dutch-specific models.
• Score: 28.99138735973746
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of public administration and the linguistic requirements of non-English contexts. We present the "Grip on LLMs" framework, a systematic evaluation suite for Dutch governmental use developed in collaboration with domain experts from a major Dutch municipal organisation. Through an advisory board process, user research, and a survey of the users of a civil-servant chatbot, we identify six evaluation dimensions (factuality, honesty, social bias, energy consumption, cost, and training data transparency) and operationalise them into a benchmark suite covering more than 30 multilingual and Dutch-specific models. Our results reveal that no single model excels across all dimensions, and that trade-offs are unavoidable: higher quality consistently comes at greater environmental impact and financial cost, while bias remains largely independent of both. We further find that factuality (whether a model answers correctly) and honesty (whether a model acknowledges what it does not know) are governed by distinct properties, with high factuality not implying high honesty. To make these findings actionable for non-technical audiences, we release a publicly accessible, user-friendly model overview designed for the full range of stakeholders involved in governmental LLM selection, from engineers to policymakers.

Related papers

• From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector [0.5154323601107853]
  We present first results of MVE, a holistic evaluation framework for the German public sector.<n>Our results reveal significant trade-offs, with no single model excelling across all dimensions.<n> evaluations should also reflect the governance requirements of the deployment context.
  arXiv  Detail & Related papers  (2026-08-18T14:28:33Z)
• Economic Evaluations of Language Models [81.69183693497281]
  EconEvals is an open-source evaluation suite to measure capabilities relevant to tasks, work activities, and occupations in the US labor economy.<n>Our evaluations improve coverage over OpenAI's GDPval benchmark, which is the existing state-of-the-art that covers 5% of US occupations, at 500x lower cost.<n>Our estimates indicate that current models could save workers substantial time on at least half of their tasks in 47% of occupations.
  arXiv  Detail & Related papers  (2026-06-26T22:26:40Z)
• MÖVE: A Holistic LLM Benchmark for the German Public Sector [0.5154323601107853]
  We present MVE (Modelle fr die ffentliche Verwaltung Evaluieren), a holistic benchmark for evaluating large language models (LLMs) in the German public sector.<n>MVE addresses these gaps by evaluating 39 models across two complementary dimensions.<n>We employ a multi-metric evaluation strategy combining classical NLP metrics, embedding-based methods, and LLM-as-a-judge approaches.
  arXiv  Detail & Related papers  (2026-06-11T09:37:45Z)
• The CitizenQuery Benchmark: A Novel Dataset and Evaluation Pipeline for Measuring LLM Performance in Citizen Query Tasks [8.50465147895087]
  "Citizen queries" are questions asked by an individual about government policies, guidance, and services that are relevant to their circumstances.<n>This represents a compelling use case for Large Language Models that respond to citizen queries with information adapted to a user's context.<n>We introduce CitizenQuery-UK, a benchmark dataset of 22 thousand pairs of citizen queries and responses.
  arXiv  Detail & Related papers  (2026-02-03T22:58:09Z)
• Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection [64.75447949495307]
  Large language models (LLMs) have been widely applied across various domains of finance.<n> behavioral biases can lead to instability and uncertainty in decision-making.<n>mfmdscen is a benchmark for evaluating behavioral biases in mfmd across diverse economic scenarios.
  arXiv  Detail & Related papers  (2026-01-08T22:00:32Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• Detecting Linguistic Bias in Government Documents Using Large language Models [2.1301560294088318]
  Existing methodologies often overlook the unique context and far-reaching impacts of governmental documents.<n>We introduce the Dutch Government Data for Bias Detection (DGDB), a dataset sourced from the Dutch House of Representatives and annotated for bias by experts.<n>We fine-tune several BERT-based models on this dataset and compare their performance with that of generative language models.
  arXiv  Detail & Related papers  (2025-02-19T08:56:16Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language.<n>This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks [68.33068005789116]
  We introduce ReDial, a benchmark containing 1.2K+ parallel query pairs in Standardized English and AAVE.<n>We evaluate widely used models, including GPT, Claude, Llama, Mistral, and the Phi model families.<n>Our work establishes a systematic and objective framework for analyzing LLM bias in dialectal queries.
  arXiv  Detail & Related papers  (2024-10-14T18:44:23Z)
• Assessing Political Bias in Large Language Models [0.624709220163167]
  We evaluate the political bias of open-source Large Language Models (LLMs) concerning political issues within the European Union (EU) from a German voter's perspective. We show that larger models, such as Llama3-70B, tend to align more closely with left-leaning political parties, while smaller models often remain neutral.
  arXiv  Detail & Related papers  (2024-05-17T15:30:18Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)
• Style Over Substance: Evaluation Biases for Large Language Models [17.13064447978519]
  This study investigates the behavior of crowd-sourced and expert annotators, as well as large language models (LLMs) Our findings reveal a concerning bias in the evaluation process, as answers with factual errors are rated more favorably than answers that are too short or contained grammatical errors. We propose independently evaluating machine-generated text across multiple dimensions, rather than merging all the evaluation aspects into a single score.
  arXiv  Detail & Related papers  (2023-07-06T14:42:01Z)
• Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs [65.9077733300329]
  Large Language Models (LLMs) have the potential to greatly enhance the analysis of public affairs documents. LLMs can be of great use to process domain-specific documents, such as those in the domain of public affairs.
  arXiv  Detail & Related papers  (2023-06-05T13:35:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.