Qalb: Largest State-of-the-Art Urdu Large Language Model for 230M Speakers with Systematic Continued Pre-training

• URL: http://arxiv.org/abs/2601.08141v1
• Date: Tue, 13 Jan 2026 02:05:05 GMT
• Title: Qalb: Largest State-of-the-Art Urdu Large Language Model for 230M Speakers with Systematic Continued Pre-training
• Authors: Muhammad Taimoor Hassan, Jawad Ahmed, Muhammad Awais,
• Abstract summary: Urdu-a language spoken by over 230 million people-remains critically underrepresented in modern NLP systems.<n>We introduce Qalb, an Urdu language model developed through a two-stage approach: continued pre-training followed by supervised fine-tuning.<n>Our results demonstrate that continued pre-training on diverse, high-quality language data, combined with targeted instruction fine-tuning, effectively adapts foundation models to low-resource languages.
• Score: 3.950299047992185
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Despite remarkable progress in large language models, Urdu-a language spoken by over 230 million people-remains critically underrepresented in modern NLP systems. Existing multilingual models demonstrate poor performance on Urdu-specific tasks, struggling with the language's complex morphology, right-to-left Nastaliq script, and rich literary traditions. Even the base LLaMA-3.1 8B-Instruct model shows limited capability in generating fluent, contextually appropriate Urdu text. We introduce Qalb, an Urdu language model developed through a two-stage approach: continued pre-training followed by supervised fine-tuning. Starting from LLaMA 3.1 8B, we perform continued pre-training on a dataset of 1.97 billion tokens. This corpus comprises 1.84 billion tokens of diverse Urdu text-spanning news archives, classical and contemporary literature, government documents, and social media-combined with 140 million tokens of English Wikipedia data to prevent catastrophic forgetting. We then fine-tune the resulting model on the Alif Urdu-instruct dataset. Through extensive evaluation on Urdu-specific benchmarks, Qalb demonstrates substantial improvements, achieving a weighted average score of 90.34 and outperforming the previous state-of-the-art Alif-1.0-Instruct model (87.1) by 3.24 points, while also surpassing the base LLaMA-3.1 8B-Instruct model by 44.64 points. Qalb achieves state-of-the-art performance with comprehensive evaluation across seven diverse tasks including Classification, Sentiment Analysis, and Reasoning. Our results demonstrate that continued pre-training on diverse, high-quality language data, combined with targeted instruction fine-tuning, effectively adapts foundation models to low-resource languages.

Related papers

• UrduLM: A Resource-Efficient Monolingual Urdu Language Model [0.0]
  Urdu, spoken by 230 million people worldwide, lacks dedicated transformer-based language models.<n>We present UrduLM, a pretrained Urdu monolingual language model trained in low-resource settings.<n>In few-shot evaluations, UrduLM achieves competitive performance with multilingual models up to 30x its size.
  arXiv  Detail & Related papers  (2026-01-25T02:49:09Z)
• Alif: Advancing Urdu Large Language Models via Multilingual Synthetic Data Distillation [2.2358141175019184]
  We propose Alif-1.0-8B-Instruct, a multilingual Urdu-English model.<n>We train the model on a high-quality, multilingual synthetic dataset (Urdu-Instruct)<n>This dataset incorporates Urdu-native chain-of-thought based reasoning, bilingual translation, cultural relevance, and ethical safety alignments.<n>Alif-1.0-8B-Instruct, built upon the pretrained Llama-3.1-8B, demonstrates superior performance compared to Llama-3.1-8B-Instruct for Urdu specific-tasks.
  arXiv  Detail & Related papers  (2025-10-10T06:41:02Z)
• UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu [12.952822154200497]
  We present the Urdu Benchmark of Linguistic Minimal Pairs (UrBLiMP)<n>UrBLiMP comprises 5,696 minimal pairs targeting ten core syntactic phenomena.<n>A human evaluation of UrBLiMP annotations yielded a 96.10% inter-annotator agreement.
  arXiv  Detail & Related papers  (2025-08-01T18:16:37Z)
• Lugha-Llama: Adapting Large Language Models for African Languages [48.97516583523523]
  Large language models (LLMs) have achieved impressive results in a wide range of natural language applications.<n>We consider how to adapt LLMs to low-resource African languages.<n>We find that combining curated data from African languages with high-quality English educational texts results in a training mix that substantially improves the model's performance on these languages.
  arXiv  Detail & Related papers  (2025-04-09T02:25:53Z)
• UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settings [0.7874708385247353]
  This paper introduces UrduLLaMA 1.0, a model derived from the open-source Llama-3.1-8B-Instruct architecture.<n>We leverage Low-Rank Adaptation (LoRA) to fine tune the model on 41,000 Urdu instructions and approximately 50,000 English-Urdu translation pairs.
  arXiv  Detail & Related papers  (2025-02-24T08:38:21Z)
• AlcLaM: Arabic Dialectal Language Model [2.8477895544986955]
  We construct an Arabic dialectal corpus comprising 3.4M sentences gathered from social media platforms. We utilize this corpus to expand the vocabulary and retrain a BERT-based model from scratch. Named AlcLaM, our model was trained using only 13 GB of text, which represents a fraction of the data used by existing models.
  arXiv  Detail & Related papers  (2024-07-18T02:13:50Z)
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic [51.922112625469836]
  We present datasetname, the first multi-task language understanding benchmark for the Arabic language. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models.
  arXiv  Detail & Related papers  (2024-02-20T09:07:41Z)
• YAYI 2: Multilingual Open-Source Large Language Models [53.92832054643197]
  We propose YAYI 2, including both base and chat models, with 30 billion parameters. YAYI 2 is pre-trained from scratch on a multilingual corpus which contains 2.65 trillion tokens filtered by our pre-training data processing pipeline. The base model is aligned with human values through supervised fine-tuning with millions of instructions and reinforcement learning from human feedback.
  arXiv  Detail & Related papers  (2023-12-22T17:34:47Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages [16.121708272597154]
  We release the IndicSUPERB benchmark for speech recognition in 12 Indian languages. We train and evaluate different self-supervised models alongside a commonly used baseline benchmark. We show that language-specific fine-tuned models are more accurate than baseline on most of the tasks.
  arXiv  Detail & Related papers  (2022-08-24T20:14:52Z)
• Few-shot Learning with Multilingual Language Models [66.49496434282564]
  We train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages. Our largest model sets new state of the art in few-shot learning in more than 20 representative languages. We present a detailed analysis of where the model succeeds and fails, showing in particular that it enables cross-lingual in-context learning.
  arXiv  Detail & Related papers  (2021-12-20T16:52:35Z)
• Understanding by Understanding Not: Modeling Negation in Language Models [81.21351681735973]
  Negation is a core construction in natural language. We propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences. We reduce the mean top1 error rate to 4% on the negated LAMA dataset.
  arXiv  Detail & Related papers  (2021-05-07T21:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.