Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models

• URL: http://arxiv.org/abs/2505.18638v2
• Date: Thu, 29 May 2025 17:11:54 GMT
• Title: Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models
• Authors: Md. Tanzib Hosain, Rajan Das Gupta, Md. Kishor Morol,
• Abstract summary: We provide a dataset of parallel Dzongkha and English test questions for Bhutanese middle and high school students.<n>The over 5K questions in our collection span a variety of scientific topics and include factual, application, and reasoning-based questions.<n>We use our parallel dataset to test a number of Large Language Models (LLMs) and find a significant performance difference between the models in English and Dzongkha.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this work, we provide DZEN, a dataset of parallel Dzongkha and English test questions for Bhutanese middle and high school students. The over 5K questions in our collection span a variety of scientific topics and include factual, application, and reasoning-based questions. We use our parallel dataset to test a number of Large Language Models (LLMs) and find a significant performance difference between the models in English and Dzongkha. We also look at different prompting strategies and discover that Chain-of-Thought (CoT) prompting works well for reasoning questions but less well for factual ones. We also find that adding English translations enhances the precision of Dzongkha question responses. Our results point to exciting avenues for further study to improve LLM performance in Dzongkha and, more generally, in low-resource languages. We release the dataset at: https://github.com/kraritt/llm_dzongkha_evaluation.

Related papers

• GanitBench: A bi-lingual benchmark for evaluating mathematical reasoning in Vision Language Models [0.0]
  GanitBench is a benchmark consisting of 1527 vision-only questions covering several topics in Mathematics.<n>We evaluate two closed source models for the same, in zero-shot Chain-of-Thought (CoT) and two-shot CoT settings.<n>GPT-4o mini is found to be the more dominant model on the benchmark, with it's highest average accuracy being 38.15%.
  arXiv  Detail & Related papers  (2025-07-31T18:24:05Z)
• TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa [13.107551474252379]
  We present TyDi QA-WANA, a question-answering dataset consisting of 28K examples divided among 10 language varieties of western Asia and northern Africa.<n>The data collection process was designed to elicit information-seeking questions, where the asker is genuinely curious to know the answer.
  arXiv  Detail & Related papers  (2025-07-23T17:20:28Z)
• Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents [7.967320126793103]
  The study fine-tunes state-of-the-art models for Sanskrit's linguistic nuances.<n>It adapts summarization techniques for Sanskrit documents to improve QA processing.<n>A dataset of 3,400 English-Sanskrit query-document pairs underpins the study.
  arXiv  Detail & Related papers  (2025-05-26T04:23:21Z)
• Lugha-Llama: Adapting Large Language Models for African Languages [48.97516583523523]
  Large language models (LLMs) have achieved impressive results in a wide range of natural language applications.<n>We consider how to adapt LLMs to low-resource African languages.<n>We find that combining curated data from African languages with high-quality English educational texts results in a training mix that substantially improves the model's performance on these languages.
  arXiv  Detail & Related papers  (2025-04-09T02:25:53Z)
• INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages [25.402797722575805]
  Indic QA Benchmark is a dataset for context grounded question answering in 11 major Indian languages.<n> Evaluations revealed weak performance in low resource languages due to a strong English language bias in their training data.<n>We also investigated the Translate Test paradigm,where inputs are translated to English for processing and the results are translated back into the source language for output.
  arXiv  Detail & Related papers  (2024-07-18T13:57:16Z)
• CaLMQA: Exploring culturally specific long-form question answering across 23 languages [58.18984409715615]
  CaLMQA is a collection of 1.5K culturally specific questions spanning 23 languages and 51 culturally translated questions from English into 22 other languages. We collect naturally-occurring questions from community web forums and hire native speakers to write questions to cover under-studied languages such as Fijian and Kirundi. Our dataset contains diverse, complex questions that reflect cultural topics (e.g. traditions, laws, news) and the language usage of native speakers.
  arXiv  Detail & Related papers  (2024-06-25T17:45:26Z)
• MahaSQuAD: Bridging Linguistic Divides in Marathi Question-Answering [0.4194295877935868]
  This research endeavors to bridge the gap of the absence of efficient QnA datasets in low-resource languages. We introduce MahaSQuAD, the first-ever full SQuAD dataset for the Indic language Marathi, consisting of 118,516 training, 11,873 validation, and 11,803 test samples.
  arXiv  Detail & Related papers  (2024-04-20T12:16:35Z)
• BEnQA: A Question Answering and Reasoning Benchmark for Bengali and English [18.217122567176585]
  We introduce BEnQA, a dataset comprising parallel Bengali and English exam questions for middle and high school levels in Bangladesh. Our dataset consists of approximately 5K questions covering several subjects in science with different types of questions, including factual, application, and reasoning-based questions. We benchmark several Large Language Models (LLMs) with our parallel dataset and observe a notable performance disparity between the models in Bengali and English.
  arXiv  Detail & Related papers  (2024-03-16T11:27:42Z)
• Question Translation Training for Better Multilingual Reasoning [108.10066378240879]
  Large language models show compelling performance on reasoning tasks but they tend to perform much worse in languages other than English. A typical solution is to translate instruction data into all languages of interest, and then train on the resulting multilingual data, which is called translate-training. In this paper we explore the benefits of question alignment, where we train the model to translate reasoning questions into English by finetuning on X-English parallel question data.
  arXiv  Detail & Related papers  (2024-01-15T16:39:10Z)
• The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants [80.4837840962273]
  We present Belebele, a dataset spanning 122 language variants. This dataset enables the evaluation of text models in high-, medium-, and low-resource languages.
  arXiv  Detail & Related papers  (2023-08-31T17:43:08Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• Ensemble Transfer Learning for Multilingual Coreference Resolution [60.409789753164944]
  A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. We design a simple but effective ensemble-based framework that combines various transfer learning techniques. We also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.
  arXiv  Detail & Related papers  (2023-01-22T18:22:55Z)
• Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval [51.004601358498135]
  Mr. TyDi is a benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages. The goal of this resource is to spur research in dense retrieval techniques in non-English languages.
  arXiv  Detail & Related papers  (2021-08-19T16:53:43Z)
• Claim Matching Beyond English to Scale Global Fact-Checking [5.836354423653351]
  We construct a novel dataset of WhatsApp tipline and public group messages alongside fact-checked claims. Our dataset contains content in high-resource (English, Hindi) and lower-resource (Bengali, Malayalam, Tamil) languages. We train our own embedding model using knowledge distillation and a high-quality "teacher" model in order to address the imbalance in embedding quality between the low- and high-resource languages.
  arXiv  Detail & Related papers  (2021-06-01T23:28:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.