GaelEval: Benchmarking LLM Performance for Scottish Gaelic

• URL: http://arxiv.org/abs/2604.02135v1
• Date: Thu, 02 Apr 2026 15:09:18 GMT
• Title: GaelEval: Benchmarking LLM Performance for Scottish Gaelic
• Abstract summary: We introduce GaelEval, the first multi-dimensional benchmark for Gaelic.<n>We find that Gemini 3 Pro Preview achieves $83.3%$ accuracy on the linguistic task, surpassing the human baseline.<n>On the cultural task, leading models exceed $90%$ accuracy, though most systems perform worse under Gaelic prompting.
• Score: 6.165315297421253
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multilingual large language models (LLMs) often exhibit emergent 'shadow' capabilities in languages without official support, yet their performance on these languages remains uneven and under-measured. This is particularly acute for morphosyntactically rich minority languages such as Scottish Gaelic, where translation benchmarks fail to capture structural competence. We introduce GaelEval, the first multi-dimensional benchmark for Gaelic, comprising: (i) an expert-authored morphosyntactic MCQA task; (ii) a culturally grounded translation benchmark and (iii) a large-scale cultural knowledge Q&A task. Evaluating 19 LLMs against a fluent-speaker human baseline ($n=30$), we find that Gemini 3 Pro Preview achieves $83.3\%$ accuracy on the linguistic task, surpassing the human baseline ($78.1\%$). Proprietary models consistently outperform open-weight systems, and in-language (Gaelic) prompting yields a small but stable advantage (+$2.4\%$). On the cultural task, leading models exceed $90\%$ accuracy, though most systems perform worse under Gaelic prompting and absolute scores are inflated relative to the manual benchmark. Overall, GaelEval reveals that frontier models achieve above-human performance on several dimensions of Gaelic grammar, demonstrates the effect of Gaelic prompting and shows a consistent performance gap favouring proprietary over open-weight models.

Related papers

• The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models [51.61416200800499]
  GaoYao is a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas.<n>First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers.<n>Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages.<n>Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs.
  arXiv  Detail & Related papers  (2026-04-22T06:19:46Z)
• Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish [12.286855282078305]
  GPT-4o, GPT-4, Claude3.5Sonnet, LLaMA3.1, MistralLarge2, LLaMA-2Chat13B, and Mistral7BInstruct are evaluated.<n>Our benchmark spans four diverse tasks: open-domain question answering, document summarization, English-to-X translation, and culturally grounded dialogue.
  arXiv  Detail & Related papers  (2025-11-05T22:09:53Z)
• Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks [33.2185998586144]
  This study benchmarks the performance of multilingual and monolingual Large Language Models (LLMs) across Arabic, English, and Indic languages.<n>Findings show significant performance differences driven by linguistic diversity and resource availability.<n> Quantization (4-bit and 8-bit) is effective in maintaining model accuracy while promoting efficiency, but aggressive pruning significantly compromises performance.
  arXiv  Detail & Related papers  (2025-07-25T22:35:10Z)
• Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters [53.59868121093848]
  We introduce Seed-X, a family of open-source language models (LLMs) with 7B parameter size.<n>The base model is pre-trained on a diverse, high-quality dataset encompassing both monolingual and bilingual content across 28 languages.<n>The instruct model is then finetuned to translate by Chain-of-Thought (CoT) reasoning and further enhanced through reinforcement learning (RL) to achieve better generalization across diverse language pairs.
  arXiv  Detail & Related papers  (2025-07-18T03:19:43Z)
• NMIXX: Domain-Adapted Neural Embeddings for Cross-Lingual eXploration of Finance [34.00870877634332]
  General-purpose sentence embedding models often struggle to capture specialized financial semantics.<n>We introduce NMIXX, a suite of cross-lingual embedding models fine-tuned with 18.8K high-confidence triplets.<n>We also release KorFinSTS, a benchmark spanning news, disclosures, research reports, and regulations.
  arXiv  Detail & Related papers  (2025-07-13T12:14:57Z)
• IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models [18.083861654053585]
  This paper introduces IrokoBench -- a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages.<n>We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings(where test sets are translated into English) across 10 open and six proprietary language models.<n>We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63% of the best-performing proprietary model GPT-4o performance.
  arXiv  Detail & Related papers  (2024-06-05T15:23:08Z)
• The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights [108.40766216456413]
  We propose a question alignment framework to bridge the gap between large language models' English and non-English performance. Experiment results show it can boost multilingual performance across diverse reasoning scenarios, model families, and sizes. We analyze representation space, generated response and data scales, and reveal how question translation training strengthens language alignment within LLMs.
  arXiv  Detail & Related papers  (2024-05-02T14:49:50Z)
• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations [59.056367787688146]
  This paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. We construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages. By utilizing translation, we construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages.
  arXiv  Detail & Related papers  (2023-10-31T08:09:20Z)
• Extrapolating Large Language Models to Non-English by Aligning Languages [109.09051737966178]
  Existing large language models show disparate capability across different languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages.
  arXiv  Detail & Related papers  (2023-08-09T13:32:06Z)
• Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages [48.28540903568198]
  We show that multilinguality is critical to making unsupervised systems practical for low-resource settings. We present a single model for 5 low-resource languages (Gujarati, Kazakh, Nepali, Sinhala, and Turkish) to and from English directions. We outperform all current state-of-the-art unsupervised baselines for these languages, achieving gains of up to 14.4 BLEU.
  arXiv  Detail & Related papers  (2020-09-23T15:07:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.