KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding

• URL: http://arxiv.org/abs/2607.17173v1
• Date: Sun, 19 Jul 2026 10:17:18 GMT
• Title: KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding
• Abstract summary: This work reports the first systematic and large-scale evaluation of large language models (LLMs) in Kyrgyz using the KyrgyzLLM-Bench benchmark suite.<n>We evaluate 26 open- and closed-source LLMs under zero-shot and few-shot settings, analyzing model performance, cross-lingual transfer, and the impact of translation artifacts on evaluation reliability.
• Score: 6.2797977024234255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating large language models (LLMs) across languages remains challenging, as most multilingual benchmarks rely on translated English datasets, often obscuring linguistic and cultural specificity in the target language. This issue is particularly pronounced for less-resourced languages such as Kyrgyz, where reliable natively authored evaluation data are scarce. Building on previously introduced Kyrgyz-language evaluation datasets, this work reports the first systematic and large-scale evaluation of LLMs in Kyrgyz using the KyrgyzLLM-Bench benchmark suite. KyrgyzLLM-Bench comprises two natively authored datasets$-$KyrgyzMMLU and KyrgyzRC$-$together with carefully translated and manually post-edited versions of WinoGrande, HellaSwag, BoolQ, and TruthfulQA. We evaluate 26 open- and closed-source LLMs under zero-shot and few-shot settings, analyzing model performance, cross-lingual transfer, and the impact of translation artifacts on evaluation reliability. Across families and tasks, model rankings transfer broadly from English to Kyrgyz on WinoGrande and BoolQ, and to a lesser extent on MMLU, while HellaSwag exhibits a substantial English-Kyrgyz performance gap consistent with translation-induced plausibility shifts. Few-shot prompting improves several open-source models on reading comprehension but behaves inconsistently for proprietary models on translated tasks. We publicly release all datasets, evaluation code, and per-model results, and integrate the Kyrgyz tasks into a widely used multilingual evaluation framework to support future research on Kyrgyz NLP.

Related papers

• MameLoshnLM: Yiddish Language Model and Evaluation Benchmark [62.936233688546984]
  We present MameLoshnLM, the first open-source 8B- parameter language model built specifically for Yiddish.<n>Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling.<n>Our results provide both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.
  arXiv  Detail & Related papers  (2026-08-06T10:24:08Z)
• Lost in Translation: Do LVLM Judges Generalize Across Languages? [46.119587015038746]
  We introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation.<n> MM-JudgeBench includes over 60K pairwise preference instances spanning 25 typologically diverse languages.<n>By evaluating 22 LVLMs, we uncover substantial cross-lingual performance variance in our proposed benchmark.
  arXiv  Detail & Related papers  (2026-04-21T12:29:10Z)
• BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR [0.06363400715351396]
  This work presents a Bangla IR dataset constructed using a BETA-labeling framework.<n>We examine whether IR datasets from other low-resource languages can be effectively reused through one-hop machine translation.
  arXiv  Detail & Related papers  (2026-02-16T06:04:04Z)
• KyrgyzBERT: A Compact, Efficient Language Model for Kyrgyz NLP [0.0]
  We introduce KyrgyzBERT, the first publicly available monolingual BERT-based language model for Kyrgyz.<n>The model has 35.9M parameters and uses a custom tokenizer designed for the language's morphological structure.<n>Kyrgyz-sst2 is a sentiment analysis benchmark built by translating the Stanford Sentiment Treebank and manually annotating the full test set.
  arXiv  Detail & Related papers  (2025-11-25T11:05:53Z)
• skLEP: A Slovak General Language Understanding Benchmark [0.030113849517062304]
  skLEP is the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding (NLU) models.<n>To create this benchmark, we curated new, original datasets tailored for Slovak and meticulously translated English NLU resources.<n>Within this paper, we also present the first systematic and extensive evaluation of a wide array of Slovak-specific, multilingual, and English pre-trained language models.
  arXiv  Detail & Related papers  (2025-06-26T17:35:04Z)
• GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models [24.450836403716547]
  GlotEval is a lightweight framework designed for massively multilingual evaluation.<n>It supports seven key tasks (machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, and intrinsic evaluation) spanning over dozens to hundreds of languages.<n>It enables a precise diagnosis of model strengths and weaknesses in diverse linguistic contexts.
  arXiv  Detail & Related papers  (2025-04-05T12:30:58Z)
• Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization [108.6908427615402]
  Cross-lingual summarization ( CLS) aims to generate a summary for the source text in a different target language.<n>Currently, instruction-tuned large language models (LLMs) excel at various English tasks.<n>Recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings.
  arXiv  Detail & Related papers  (2024-10-26T00:39:44Z)
• Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages [51.301942056881146]
  We investigate how large language models (LLMs) function as rerankers in cross-lingual information retrieval systems for African languages. Our implementation covers English and four African languages (Hausa, Somali, Swahili, and Yoruba) We examine cross-lingual reranking with queries in English and passages in the African languages.
  arXiv  Detail & Related papers  (2023-12-26T18:38:54Z)
• The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants [80.4837840962273]
  We present Belebele, a dataset spanning 122 language variants. This dataset enables the evaluation of text models in high-, medium-, and low-resource languages.
  arXiv  Detail & Related papers  (2023-08-31T17:43:08Z)
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation [105.41167108465085]
  Cross-lingual Machine Reading (CLMRC) remains a challenging problem due to the lack of large-scale datasets in low-source languages. We propose a novel augmentation approach named Language Branch Machine Reading (LBMRC) LBMRC trains multiple machine reading comprehension (MRC) models proficient in individual language. We devise a multilingual distillation approach to amalgamate knowledge from multiple language branch models to a single model for all target languages.
  arXiv  Detail & Related papers  (2020-10-27T13:12:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.