NativQA Framework: Enabling LLMs with Native, Local, and Everyday Knowledge

• URL: http://arxiv.org/abs/2504.05995v1
• Date: Tue, 08 Apr 2025 13:01:51 GMT
• Title: NativQA Framework: Enabling LLMs with Native, Local, and Everyday Knowledge
• Authors: Firoj Alam, Md Arid Hasan, Sahinur Rahman Laskar, Mucahid Kutlu, Shammur Absar Chowdhury,
• Abstract summary: We propose a framework, NativQA, that can seamlessly construct large-scale, culturally and regionally aligned QA datasets in native languages.<n>The framework has been evaluated across 39 locations in 24 countries and in 7 languages, ranging from extremely low-resource to high-resource languages.
• Score: 10.754622388103856
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The rapid advancement of large language models (LLMs) has raised concerns about cultural bias, fairness, and their applicability in diverse linguistic and underrepresented regional contexts. To enhance and benchmark the capabilities of LLMs, there is a need to develop large-scale resources focused on multilingual, local, and cultural contexts. In this study, we propose a framework, NativQA, that can seamlessly construct large-scale, culturally and regionally aligned QA datasets in native languages. The framework utilizes user-defined seed queries and leverages search engines to collect location-specific, everyday information. It has been evaluated across 39 locations in 24 countries and in 7 languages, ranging from extremely low-resource to high-resource languages, which resulted over 300K Question Answer (QA) pairs. The developed resources can be used for LLM benchmarking and further fine-tuning. The framework has been made publicly available for the community (https://gitlab.com/nativqa/nativqa-framework).

Related papers

• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet [55.39571645315926]
  Large Language Models (LLMs) rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages.<n>For low-resource languages, the limited availability of such data hampers the models' ability to generalize effectively.<n>We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages.
  arXiv  Detail & Related papers  (2025-01-31T12:23:28Z)
• LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Reasoning [28.288949710191158]
  Large language models (LLMs) have exhibited impressive multilingual reasoning capabilities, driven by extensive multilingual pre-training corpora and instruction fine-tuning data.<n>A performance gap exists between high- and low-resource language reasoning tasks due to the language imbalance in the pre-training corpus.<n>We propose LinguaLIFT, a two-stage instruction tuning framework for advancing low-resource language reasoning.
  arXiv  Detail & Related papers  (2024-12-17T03:03:17Z)
• INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge [36.234295907476515]
  The development of functional large language models (LLM) is bottlenecked by the lack of high-quality evaluation resources in languages other than English.<n>In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts.
  arXiv  Detail & Related papers  (2024-11-29T16:03:14Z)
• Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization [108.6908427615402]
  Cross-lingual summarization ( CLS) aims to generate a summary for the source text in a different target language.<n>Currently, instruction-tuned large language models (LLMs) excel at various English tasks.<n>Recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings.
  arXiv  Detail & Related papers  (2024-10-26T00:39:44Z)
• LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content [9.539308087147134]
  Large Language Models (LLMs) have demonstrated remarkable success as general-purpose task solvers across various fields. This study focuses on developing a specialized LLM, LlamaLens, for analyzing news and social media content in a multilingual context. We demonstrate that LlamaLens outperforms the current state-of-the-art (SOTA) on 23 testing sets, and achieves comparable performance on 8 sets.
  arXiv  Detail & Related papers  (2024-10-20T06:37:37Z)
• L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Context [0.4194295877935868]
  We present the L3Cube-IndicQuest, a gold-standard factual question-answering benchmark dataset. The dataset contains 200 question-answer pairs, each for English and 19 Indic languages, covering five domains specific to the Indic region.
  arXiv  Detail & Related papers  (2024-09-13T10:48:35Z)
• NativQA: Multilingual Culturally-Aligned Natural Query for LLMs [12.35947908812959]
  We propose a language-independent framework, NativQA, to seamlessly construct culturally and regionally aligned QA datasets in native languages. We demonstrate the efficacy of the proposed framework by designing a multilingual natural QA dataset, mnqa, consisting of 64k manually annotated QA pairs in seven languages. We also showcase the framework efficacy in constructing fine-tuning data especially for low-resource and dialectally-rich languages.
  arXiv  Detail & Related papers  (2024-07-13T09:34:00Z)
• Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Models [7.615938028813914]
  We studied linguistic preference in a cross-language RAG-based information search setting. We found that LLMs displayed systemic bias towards information in the same language as the query language.
  arXiv  Detail & Related papers  (2024-07-07T21:26:36Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora.<n>But can these models relate corresponding concepts across languages, i.e., be crosslingual?<n>This study evaluates state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models [18.083861654053585]
  This paper introduces IrokoBench -- a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages.<n>We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings(where test sets are translated into English) across 10 open and six proprietary language models.<n>We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63% of the best-performing proprietary model GPT-4o performance.
  arXiv  Detail & Related papers  (2024-06-05T15:23:08Z)
• LLMs Are Few-Shot In-Context Low-Resource Language Learners [59.74451570590808]
  In-context learning (ICL) empowers large language models (LLMs) to perform diverse tasks in underrepresented languages. We extensively study ICL and its cross-lingual variation (X-ICL) on 25 low-resource and 7 relatively higher-resource languages. Our study concludes the significance of few-shot in-context information on enhancing the low-resource understanding quality of LLMs.
  arXiv  Detail & Related papers  (2024-03-25T07:55:29Z)
• CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages [86.90220551111096]
  Training datasets for large language models (LLMs) are often not fully disclosed. We present CulturaX, a substantial multilingual dataset with 6.3 trillion tokens in 167 languages.
  arXiv  Detail & Related papers  (2023-09-17T23:49:10Z)
• Prix-LM: Pretraining for Multilingual Knowledge Base Construction [59.02868906044296]
  We propose a unified framework, Prix-LM, for multilingual knowledge construction and completion. We leverage two types of knowledge, monolingual triples and cross-lingual links, extracted from existing multilingual KBs. Experiments on standard entity-related tasks, such as link prediction in multiple languages, cross-lingual entity linking and bilingual lexicon induction, demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2021-10-16T02:08:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.