MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languages

• URL: http://arxiv.org/abs/2503.01150v1
• Date: Mon, 03 Mar 2025 03:56:03 GMT
• Title: MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languages
• Authors: Chen Zhang, Mingxu Tao, Zhiyuan Liao, Yansong Feng,
• Abstract summary: We introduce MiLiC-Eval, a benchmark designed for minority languages in China, featuring 24K instances across 9 tasks.<n>MiLiC-Eval focuses on underrepresented writing systems and provides a fine-grained assessment of linguistic and problem-solving skills.
• Score: 30.66853618502553
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) excel in high-resource languages but struggle with low-resource languages (LRLs), particularly those spoken by minority communities in China, such as Tibetan, Uyghur, Kazakh, and Mongolian. To systematically track the progress in these languages, we introduce MiLiC-Eval, a benchmark designed for minority languages in China, featuring 24K instances across 9 tasks. MiLiC-Eval focuses on underrepresented writing systems and provides a fine-grained assessment of linguistic and problem-solving skills. Our evaluation reveals that LLMs perform poorly on syntax-intensive tasks and multi-script languages. We further demonstrate how MiLiC-Eval can help advance LRL research in handling diverse writing systems and understanding the process of language adaptation.

Related papers

• PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts [79.84059473102778]
  PolyMath is a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels. Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation.
  arXiv  Detail & Related papers  (2025-04-25T15:39:04Z)
• INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge [36.234295907476515]
  The development of functional large language models (LLM) is bottlenecked by the lack of high-quality evaluation resources in languages other than English.<n>In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts.
  arXiv  Detail & Related papers  (2024-11-29T16:03:14Z)
• How Chinese are Chinese Language Models? The Puzzling Lack of Language Policy in China's LLMs [2.9123921488295768]
  We evaluate six open-source multilingual LLMs pre-trained by Chinese companies on 18 languages. Our experiments show Chinese LLMs performance on diverse languages is indistinguishable from international LLMs. We find no sign of any consistent policy, either for or against, language diversity in China's LLM development.
  arXiv  Detail & Related papers  (2024-07-12T19:21:40Z)
• Teaching LLMs to Abstain across Languages via Multilingual Feedback [40.84205285309612]
  We show that multilingual feedback helps identify knowledge gaps across diverse languages, cultures, and communities. Extensive experiments demonstrate that our multilingual feedback approach outperforms various strong baselines. Further analysis reveals that multilingual feedback is both an effective and a more equitable abstain strategy to serve diverse language speakers.
  arXiv  Detail & Related papers  (2024-06-22T21:59:12Z)
• A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias [5.096332588720052]
  We present an overview of MLLMs, covering their evolutions, key techniques, and multilingual capacities.<n>Thirdly, we survey the state-of-the-art studies of multilingual representations and investigate whether the current MLLMs can learn a universal language representation.<n>Fourthly, we discuss bias on MLLMs, including its categories, evaluation metrics, and debiasing techniques.
  arXiv  Detail & Related papers  (2024-04-01T05:13:56Z)
• LLMs Are Few-Shot In-Context Low-Resource Language Learners [59.74451570590808]
  In-context learning (ICL) empowers large language models (LLMs) to perform diverse tasks in underrepresented languages. We extensively study ICL and its cross-lingual variation (X-ICL) on 25 low-resource and 7 relatively higher-resource languages. Our study concludes the significance of few-shot in-context information on enhancing the low-resource understanding quality of LLMs.
  arXiv  Detail & Related papers  (2024-03-25T07:55:29Z)
• CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models [53.9835961434552]
  We introduce the Chinese Instruction-Following Benchmark (CIF-Bench) to evaluate the generalizability of large language models (LLMs) to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances. To mitigate data contamination, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance.
  arXiv  Detail & Related papers  (2024-02-20T16:02:12Z)
• Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages [51.301942056881146]
  We investigate how large language models (LLMs) function as rerankers in cross-lingual information retrieval systems for African languages. Our implementation covers English and four African languages (Hausa, Somali, Swahili, and Yoruba) We examine cross-lingual reranking with queries in English and passages in the African languages.
  arXiv  Detail & Related papers  (2023-12-26T18:38:54Z)
• MC$^2$: Towards Transparent and Culturally-Aware NLP for Minority Languages in China [33.08119305158835]
  We present MC$2$, a Multilingual Corpus of Minority Languages in China. MC$2$ includes four underrepresented languages: Tibetan, Uyghur, Kazakh, and Mongolian.
  arXiv  Detail & Related papers  (2023-11-14T17:45:50Z)
• Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts [75.33019401706188]
  Large language models (LLMs) are known to effectively perform tasks by simply observing few exemplars. We propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. Our unsupervised prompting method performs on par with supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages.
  arXiv  Detail & Related papers  (2023-06-20T08:27:47Z)
• CMMLU: Measuring massive multitask language understanding in Chinese [133.70911295934746]
  This paper introduces a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.
  arXiv  Detail & Related papers  (2023-06-15T15:49:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.