BanglaByT5: Byte-Level Modelling for Bangla

• URL: http://arxiv.org/abs/2505.17102v1
• Date: Wed, 21 May 2025 07:39:07 GMT
• Title: BanglaByT5: Byte-Level Modelling for Bangla
• Authors: Pramit Bhattacharyya, Arnab Bhattacharya,
• Abstract summary: We introduce BanglaByT5, the first byte-level encoder-decoder model explicitly tailored for Bangla.<n>Built upon a small variant of Googles ByT5 architecture, BanglaByT5 is pre-trained on a 14GB curated corpus combining high-quality literary and newspaper articles.
• Score: 3.9018931027384056
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have achieved remarkable success across various natural language processing tasks. However, most LLM models use traditional tokenizers like BPE and SentencePiece, which fail to capture the finer nuances of a morphologically rich language like Bangla (Bengali). In this work, we introduce BanglaByT5, the first byte-level encoder-decoder model explicitly tailored for Bangla. Built upon a small variant of Googles ByT5 architecture, BanglaByT5 is pre-trained on a 14GB curated corpus combining high-quality literary and newspaper articles. Through zeroshot and supervised evaluations across generative and classification tasks, BanglaByT5 demonstrates competitive performance, surpassing several multilingual and larger models. Our findings highlight the efficacy of byte-level modelling for morphologically rich languages and highlight BanglaByT5 potential as a lightweight yet powerful tool for Bangla NLP, particularly in both resource-constrained and scalable environments.

Related papers

• TigerLLM -- A Family of Bangla Large Language Models [8.258559455995917]
  We introduce TigerLLM - a family of Bangla language models.<n>Our results demonstrate that these models surpass all open-source alternatives and also outperform larger proprietary models like GPT3.5.
  arXiv  Detail & Related papers  (2025-03-14T01:41:16Z)
• Zero-Shot Multi-Label Classification of Bangla Documents: Large Decoders Vs. Classic Encoders [0.0]
  Bangla is a language spoken by over 300 million native speakers and ranked as the sixth most spoken language worldwide.<n>Our evaluation of 32 state-of-the-art models reveals that, existing so-called powerful encoders and decoders still struggle to achieve high accuracy on the Bangla Zero-Shot-MLC task.
  arXiv  Detail & Related papers  (2025-03-04T20:39:07Z)
• BongLLaMA: LLaMA for Bangla Language [0.0]
  BongLLaMA is an open-source large language model fine-tuned exclusively on large Bangla corpora and instruction-tuning datasets. We present our methodology, data augmentation techniques, fine-tuning details, and comprehensive benchmarking results showcasing the utility of BongLLaMA on BLP tasks.
  arXiv  Detail & Related papers  (2024-10-28T16:44:02Z)
• Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages [76.35234803589412]
  MPM is an effective training paradigm for training large multimodal models in non-English languages. We build large multimodal models VisCPM in image-to-text and text-to-image generation, which achieve state-of-the-art (open-source) performance in Chinese.
  arXiv  Detail & Related papers  (2023-08-23T09:55:41Z)
• mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations [54.42422445568523]
  mmT5 is a modular multilingual sequence-to-sequence model. It disentangles language-specific information from language-agnostic information. Compared to mT5, mmT5 raises the rate of generating text in the correct language under zero-shot settings from 7% to 99%.
  arXiv  Detail & Related papers  (2023-05-23T16:38:01Z)
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model [264.96498474333697]
  Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. We present BLOOM, a 176B- parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages.
  arXiv  Detail & Related papers  (2022-11-09T18:48:09Z)
• Crosslingual Generalization through Multitask Finetuning [80.8822603322471]
  Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages.
  arXiv  Detail & Related papers  (2022-11-03T13:19:32Z)
• BanglaNLG: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla [21.47743471497797]
  This work presents a benchmark for evaluating natural language generation models in Bangla. We aggregate three challenging conditional text generation tasks under the BanglaNLG benchmark. Using a clean corpus of 27.5 GB of Bangla data, we pretrain BanglaT5, a sequence-to-sequence Transformer model for Bangla. BanglaT5 achieves state-of-the-art performance in all of these tasks, outperforming mT5 (base) by up to 5.4%.
  arXiv  Detail & Related papers  (2022-05-23T06:54:56Z)
• PaLM: Scaling Language Modeling with Pathways [180.69584031908113]
  We trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks.
  arXiv  Detail & Related papers  (2022-04-05T16:11:45Z)
• BanglaBERT: Combating Embedding Barrier for Low-Resource Language Understanding [1.7000879291900044]
  We build a Bangla natural language understanding model pre-trained on 18.6 GB data we crawled from top Bangla sites on the internet. Our model outperforms multilingual baselines and previous state-of-the-art results by 1-6%. We identify a major shortcoming of multilingual models that hurt performance for low-resource languages that don't share writing scripts with any high resource one.
  arXiv  Detail & Related papers  (2021-01-01T09:28:45Z)
• mT5: A massively multilingual pre-trained text-to-text transformer [60.0210636815514]
  "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on English-language NLP tasks. We introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages.
  arXiv  Detail & Related papers  (2020-10-22T17:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.