BanglaLlama: LLaMA for Bangla Language

• URL: http://arxiv.org/abs/2410.21200v2
• Date: Mon, 06 Oct 2025 18:11:26 GMT
• Title: BanglaLlama: LLaMA for Bangla Language
• Authors: Abdullah Khan Zehady, Shubhashis Roy Dipta, Naymul Islam, Safi Al Mamun, Santu Karmaker,
• Abstract summary: Despite being the 5th largest spoken language in the world, Bangla is still a "low-resource" language.<n>Existing pretrained language models often struggle to perform well on Bangla Language Processing (BLP) tasks.<n>This paper introduces two high-quality translated Bangla-instruction datasets totaling 224k samples.
• Score: 1.0710988917914002
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bangla is a language spoken by approximately 240 million native speakers and around 300 million people worldwide. Despite being the 5th largest spoken language in the world, Bangla is still a "low-resource" language, and existing pretrained language models often struggle to perform well on Bangla Language Processing (BLP) tasks. This paper addresses this gap by: (1) introducing two high-quality translated Bangla-instruction datasets totaling 224k samples - Bangla-Orca (172k) and Bangla-Alpaca (52k); and (2) leveraging these datasets to develop BanglaLlama, an open-source family of Bangla-specific LLMs, consisting of five base and instruct variants. We present our methodology, two large datasets, and comprehensive benchmarking results showcasing the effectiveness of our dataset and model on multiple benchmarks. We believe our proposed datasets and models will serve as the new standard baseline for future research focused on this widely spoken yet "low-resource" language.

Related papers

• TigerCoder: A Novel Suite of LLMs for Code Generation in Bangla [37.210208249613]
  Despite being the 5th most spoken language, Bangla remains underrepresented in Large Language Models (LLMs)<n>This primarily stems from the scarcity of high-quality data to pre-train and/or finetune such models.<n>We offer three major contributions: (1) a comprehensive Bangla code instruction datasets for programming domain adaptation; (2) MBPP-Bangla, an evaluation benchmark for Bangla code generation; and (3) the TigerCoder-family of Code LLMs, achieving significant 11-18% performance gains at Pass@1 over existing multilingual and general-purpose Bangla LLMs.
  arXiv  Detail & Related papers  (2025-09-11T02:25:49Z)
• Improving Bangla Linguistics: Advanced LSTM, Bi-LSTM, and Seq2Seq Models for Translating Sylheti to Modern Bangla [3.11717505289722]
  Every division of Bangladesh has its own local language like Sylheti, Chittagong etc.<n>This research is for the local language and this particular paper is on Sylheti language.<n>It presented a comprehensive system using Natural Language Processing or NLP techniques for translating Pure or Modern Bangla to locally spoken Sylheti Bangla language.
  arXiv  Detail & Related papers  (2025-05-24T14:13:45Z)
• TigerLLM -- A Family of Bangla Large Language Models [8.258559455995917]
  We introduce TigerLLM - a family of Bangla language models. Our results demonstrate that these models surpass all open-source alternatives and also outperform larger proprietary models like GPT3.5.
  arXiv  Detail & Related papers  (2025-03-14T01:41:16Z)
• Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakers [80.69714909319842]
  $texttBabel$ is an open multilingual LLM that covers the top 25 languages by number of speakers.<n>It supports over 90% of the global population, and includes many languages neglected by other open multilingual LLMs.
  arXiv  Detail & Related papers  (2025-03-02T11:53:55Z)
• Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages [55.36534539177367]
  This paper introduces Pangea, a multilingual multimodal large language model (MLLM) trained on a diverse 6M instruction dataset spanning 39 languages.<n>P Pangea significantly outperforms existing open-source models in multilingual settings and diverse cultural contexts.<n>We fully open-source our data, code, and trained checkpoints, to facilitate the development of inclusive and robust multilingual MLLMs.
  arXiv  Detail & Related papers  (2024-10-21T16:19:41Z)
• MaLA-500: Massive Language Adaptation of Large Language Models [61.440556436524]
  MaLA-500 is a novel large language model designed to cover an extensive range of 534 languages. Our intrinsic evaluation demonstrates that MaLA-500 is better at predicting the given texts of low-resource languages than existing multilingual LLMs.
  arXiv  Detail & Related papers  (2024-01-24T08:57:39Z)
• BanglaNLP at BLP-2023 Task 2: Benchmarking different Transformer Models for Sentiment Analysis of Bangla Social Media Posts [0.46040036610482665]
  This paper presents our submission to Task 2 (Sentiment Analysis of Bangla Social Media Posts) of the BLP Workshop. Our quantitative results show that transfer learning really helps in better learning of the models in this low-resource language scenario. We obtain a micro-F1 of 67.02% on the test set and our performance in this shared task is ranked at 21 in the leaderboard.
  arXiv  Detail & Related papers  (2023-10-13T16:46:38Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• Baichuan 2: Open Large-scale Language Models [51.56361715162972]
  We present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval.
  arXiv  Detail & Related papers  (2023-09-19T04:13:22Z)
• PolyLM: An Open Source Polyglot Large Language Model [57.64420154135178]
  We present PolyLM, a multilingual large language model (LLMs) trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning.
  arXiv  Detail & Related papers  (2023-07-12T09:00:37Z)
• ML-SUPERB: Multilingual Speech Universal PERformance Benchmark [73.65853301350042]
  Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks. This paper presents multilingual SUPERB, covering 143 languages (ranging from high-resource to endangered), and considering both automatic speech recognition and language identification. Similar to the SUPERB benchmark, we find speech SSL models can significantly improve performance compared to FBANK features.
  arXiv  Detail & Related papers  (2023-05-18T00:01:27Z)
• On Evaluation of Bangla Word Analogies [0.8658596218544772]
  This paper presents a high-quality dataset for evaluating the quality of Bangla word embeddings. Despite being the 7th most-spoken language in the world, Bangla is a low-resource language and popular NLP models fail to perform well.
  arXiv  Detail & Related papers  (2023-04-10T14:27:35Z)
• BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting [50.24676567971536]
  The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. We apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages. We conclude that with sufficient training data language adaptation can generalize well to diverse languages.
  arXiv  Detail & Related papers  (2022-12-19T15:24:45Z)
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model [264.96498474333697]
  Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. We present BLOOM, a 176B- parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages.
  arXiv  Detail & Related papers  (2022-11-09T18:48:09Z)
• BanglaNLG: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla [21.47743471497797]
  This work presents a benchmark for evaluating natural language generation models in Bangla. We aggregate three challenging conditional text generation tasks under the BanglaNLG benchmark. Using a clean corpus of 27.5 GB of Bangla data, we pretrain BanglaT5, a sequence-to-sequence Transformer model for Bangla. BanglaT5 achieves state-of-the-art performance in all of these tasks, outperforming mT5 (base) by up to 5.4%.
  arXiv  Detail & Related papers  (2022-05-23T06:54:56Z)
• A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models [2.5768647103950357]
  We provide a review of Bangla NLP tasks, resources, and tools available to the research community. We benchmark datasets collected from various platforms for nine NLP tasks using current state-of-the-art algorithms. We report our results using both individual and consolidated datasets and provide data for future research.
  arXiv  Detail & Related papers  (2021-07-08T13:49:46Z)
• BanglaBERT: Combating Embedding Barrier for Low-Resource Language Understanding [1.7000879291900044]
  We build a Bangla natural language understanding model pre-trained on 18.6 GB data we crawled from top Bangla sites on the internet. Our model outperforms multilingual baselines and previous state-of-the-art results by 1-6%. We identify a major shortcoming of multilingual models that hurt performance for low-resource languages that don't share writing scripts with any high resource one.
  arXiv  Detail & Related papers  (2021-01-01T09:28:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.