From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector

• URL: http://arxiv.org/abs/2608.17827v1
• Date: Tue, 18 Aug 2026 14:28:33 GMT
• Title: From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector
• Abstract summary: We present first results of MVE, a holistic evaluation framework for the German public sector.<n>Our results reveal significant trade-offs, with no single model excelling across all dimensions.<n> evaluations should also reflect the governance requirements of the deployment context.
• Score: 0.5154323601107853
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Public institutions face a persistent challenge in selecting LLMs suited to their specific context. Existing benchmarks, however, are of limited use as they primarily reflect English-language and US-centric settings, and often only evaluate task performance. In this paper, we present first results of MÖVE, a holistic evaluation framework for the German public sector, examining three rarely considered governance dimensions: energy consumption, provider transparency, and knowledge of German-party positions. Our results reveal significant trade-offs, with no single model excelling across all dimensions: estimated energy consumption varies more than 60-fold and is not explained by model size alone, information disclosure varies systematically across providers, and European models do not exhibit stronger knowledge of German party positions. Model selection for public institutions thus cannot rely on performance rankings alone. Instead, evaluations should also reflect the governance requirements of the deployment context.

Related papers

• From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch [28.99138735973746]
  "Grip on LLMs" is a systematic evaluation suite for Dutch governmental use.<n>We identify six evaluation dimensions (factuality, honesty, social bias, energy consumption, cost, and training data transparency)<n>We operationalise them into a benchmark suite covering more than 30 multilingual and Dutch-specific models.
  arXiv  Detail & Related papers  (2026-08-10T17:59:05Z)
• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• MÖVE: A Holistic LLM Benchmark for the German Public Sector [0.5154323601107853]
  We present MVE (Modelle fr die ffentliche Verwaltung Evaluieren), a holistic benchmark for evaluating large language models (LLMs) in the German public sector.<n>MVE addresses these gaps by evaluating 39 models across two complementary dimensions.<n>We employ a multi-metric evaluation strategy combining classical NLP metrics, embedding-based methods, and LLM-as-a-judge approaches.
  arXiv  Detail & Related papers  (2026-06-11T09:37:45Z)
• "As Eastern Powers, I will veto." : An Investigation of Nation-level Bias of Large Language Models in International Relations [7.585969077788284]
  This paper systematically examines nation-level biases exhibited by Large Language Models (LLMs) within the domain of International Relations (IR)<n>We develop a bias evaluation framework comprising three distinct tests to explore nation-level bias in various LLMs.<n>We introduce a debiasing framework that improves LLMs' factual reasoning combining Retrieval-Augmented Generation with Reflexion-based self-reflection techniques.
  arXiv  Detail & Related papers  (2025-11-12T08:42:54Z)
• HALF: Harm-Aware LLM Fairness Evaluation Aligned with Deployment [52.374772443536045]
  HALF (Harm-Aware LLM Fairness) is a framework that assesses model bias in realistic applications and weighs the outcomes by harm severity.<n>We show that HALF exposes a clear gap between previous benchmarking success and deployment readiness.
  arXiv  Detail & Related papers  (2025-10-14T07:13:26Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• Pitfalls of Evaluating Language Models with Open Benchmarks [5.387340038770958]
  This study exposes these weaknesses by systematically constructing cheating'' models on public test sets.<n>High leaderboard performance on open benchmarks may not always reflect real-world effectiveness.<n>A fundamental reevaluation of current benchmarking practices is essential to ensure robust and trustworthy LM assessments.
  arXiv  Detail & Related papers  (2025-07-01T06:17:48Z)
• CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models [58.57987316300529]
  Large Language Models (LLMs) are increasingly deployed to handle various natural language processing (NLP) tasks.<n>To evaluate the biases exhibited by LLMs, researchers have recently proposed a variety of datasets.<n>We propose CEB, a Compositional Evaluation Benchmark that covers different types of bias across different social groups and tasks.
  arXiv  Detail & Related papers  (2024-07-02T16:31:37Z)
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models [94.31327813151208]
  BiGGen Bench is a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks.<n>A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation.
  arXiv  Detail & Related papers  (2024-06-09T12:30:30Z)
• ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Models [17.562961249150295]
  We propose the ZhuJiu benchmark for large language models (LLMs) evaluation. ZhuJiu is the pioneering benchmark that fully assesses LLMs in Chinese, while also providing equally robust evaluation abilities in English. The ZhuJiu benchmark and open-participation leaderboard are publicly released at http://www.zhujiu-benchmark.com/.
  arXiv  Detail & Related papers  (2023-08-28T06:56:44Z)
• LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models [55.60306377044225]
  Large language models (LLMs) have enabled impressive zero-shot capabilities across various natural language tasks. This paper explores two options for exploiting the emergent abilities of LLMs for zero-shot NLG assessment. For moderate-sized open-source LLMs, such as FlanT5 and Llama2-chat, comparative assessment is superior to prompt scoring.
  arXiv  Detail & Related papers  (2023-07-15T22:02:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.