BongLLaMA: LLaMA for Bangla Language

• URL: http://arxiv.org/abs/2410.21200v1
• Date: Mon, 28 Oct 2024 16:44:02 GMT
• Title: BongLLaMA: LLaMA for Bangla Language
• Authors: Abdullah Khan Zehady, Safi Al Mamun, Naymul Islam, Santu Karmaker,
• Abstract summary: BongLLaMA is an open-source large language model fine-tuned exclusively on large Bangla corpora and instruction-tuning datasets. We present our methodology, data augmentation techniques, fine-tuning details, and comprehensive benchmarking results showcasing the utility of BongLLaMA on BLP tasks.
• Score: 0.0
• License:
• Abstract: Bangla (or "Bengali") is a language spoken by approximately 240 million native speakers and around 300 million people worldwide. Despite being the 5th largest spoken language in the world, Bangla is still a "low-resource" language, and existing pretrained language models often struggle to perform well on Bangla Language Processing (BLP) tasks. This work addresses this gap by introducing BongLLaMA (i.e., Bangla-LLaMA), an open-source large language model fine-tuned exclusively on large Bangla corpora and instruction-tuning datasets. We present our methodology, data augmentation techniques, fine-tuning details, and comprehensive benchmarking results showcasing the utility of BongLLaMA on BLP tasks. We believe BongLLaMA will serve as the new standard baseline for Bangla Language Models and, thus, facilitate future benchmarking studies focused on this widely-spoken yet "low-resource" language. All BongLLaMA models are available for public use at https://huggingface.co/BanglaLLM.

Related papers

• MaLA-500: Massive Language Adaptation of Large Language Models [61.440556436524]
  MaLA-500 is a novel large language model designed to cover an extensive range of 534 languages. Our intrinsic evaluation demonstrates that MaLA-500 is better at predicting the given texts of low-resource languages than existing multilingual LLMs.
  arXiv  Detail & Related papers  (2024-01-24T08:57:39Z)
• BanglaNLP at BLP-2023 Task 2: Benchmarking different Transformer Models for Sentiment Analysis of Bangla Social Media Posts [0.46040036610482665]
  This paper presents our submission to Task 2 (Sentiment Analysis of Bangla Social Media Posts) of the BLP Workshop. Our quantitative results show that transfer learning really helps in better learning of the models in this low-resource language scenario. We obtain a micro-F1 of 67.02% on the test set and our performance in this shared task is ranked at 21 in the leaderboard.
  arXiv  Detail & Related papers  (2023-10-13T16:46:38Z)
• Baichuan 2: Open Large-scale Language Models [51.56361715162972]
  We present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval.
  arXiv  Detail & Related papers  (2023-09-19T04:13:22Z)
• PolyLM: An Open Source Polyglot Large Language Model [57.64420154135178]
  We present PolyLM, a multilingual large language model (LLMs) trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning.
  arXiv  Detail & Related papers  (2023-07-12T09:00:37Z)
• ML-SUPERB: Multilingual Speech Universal PERformance Benchmark [73.65853301350042]
  Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks. This paper presents multilingual SUPERB, covering 143 languages (ranging from high-resource to endangered), and considering both automatic speech recognition and language identification. Similar to the SUPERB benchmark, we find speech SSL models can significantly improve performance compared to FBANK features.
  arXiv  Detail & Related papers  (2023-05-18T00:01:27Z)
• On Evaluation of Bangla Word Analogies [0.8658596218544772]
  This paper presents a high-quality dataset for evaluating the quality of Bangla word embeddings. Despite being the 7th most-spoken language in the world, Bangla is a low-resource language and popular NLP models fail to perform well.
  arXiv  Detail & Related papers  (2023-04-10T14:27:35Z)
• BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting [50.24676567971536]
  The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. We apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages. We conclude that with sufficient training data language adaptation can generalize well to diverse languages.
  arXiv  Detail & Related papers  (2022-12-19T15:24:45Z)
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model [264.96498474333697]
  Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. We present BLOOM, a 176B- parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages.
  arXiv  Detail & Related papers  (2022-11-09T18:48:09Z)
• BanglaNLG: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla [21.47743471497797]
  This work presents a benchmark for evaluating natural language generation models in Bangla. We aggregate three challenging conditional text generation tasks under the BanglaNLG benchmark. Using a clean corpus of 27.5 GB of Bangla data, we pretrain BanglaT5, a sequence-to-sequence Transformer model for Bangla. BanglaT5 achieves state-of-the-art performance in all of these tasks, outperforming mT5 (base) by up to 5.4%.
  arXiv  Detail & Related papers  (2022-05-23T06:54:56Z)
• A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models [2.5768647103950357]
  We provide a review of Bangla NLP tasks, resources, and tools available to the research community. We benchmark datasets collected from various platforms for nine NLP tasks using current state-of-the-art algorithms. We report our results using both individual and consolidated datasets and provide data for future research.
  arXiv  Detail & Related papers  (2021-07-08T13:49:46Z)
• BanglaBERT: Combating Embedding Barrier for Low-Resource Language Understanding [1.7000879291900044]
  We build a Bangla natural language understanding model pre-trained on 18.6 GB data we crawled from top Bangla sites on the internet. Our model outperforms multilingual baselines and previous state-of-the-art results by 1-6%. We identify a major shortcoming of multilingual models that hurt performance for low-resource languages that don't share writing scripts with any high resource one.
  arXiv  Detail & Related papers  (2021-01-01T09:28:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.