Tamil-Llama: A New Tamil Language Model Based on Llama 2

• URL: http://arxiv.org/abs/2311.05845v1
• Date: Fri, 10 Nov 2023 03:02:39 GMT
• Title: Tamil-Llama: A New Tamil Language Model Based on Llama 2
• Authors: Abhinand Balachandran
• Abstract summary: This paper enhances the open-source LLaMA model with an addition of 16,000 Tamil tokens, aiming to achieve superior text generation and comprehension in Tamil language. We strategically employ the LoRA methodology for efficient model training on a comprehensive Tamil corpus, ensuring computational feasibility and model robustness. Our results showcase significant performance improvements in Tamil text generation, with potential implications for the broader landscape of Large Language Models in Indian languages.
• Score: 6.449795539095749
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language modeling has witnessed remarkable advancements in recent years, with Large Language Models (LLMs) like ChatGPT setting unparalleled benchmarks in human-like text generation. However, a prevailing limitation is the underrepresentation of languages like Tamil in these cutting-edge models, leading to suboptimal performance in diverse linguistic contexts. This paper addresses this lacuna, enhancing the open-source LLaMA model with an addition of 16,000 Tamil tokens, aiming to achieve superior text generation and comprehension in the Tamil language. We strategically employ the LoRA methodology for efficient model training on a comprehensive Tamil corpus, ensuring computational feasibility and model robustness. Moreover, we introduce a Tamil-translated version of the Alpaca dataset and a subset of the OpenOrca dataset tailored for instruction fine-tuning. Our results showcase significant performance improvements in Tamil text generation, with potential implications for the broader landscape of LLMs in Indian languages. We further underscore our commitment to open research by making our models, datasets, and code publicly accessible, fostering further innovations in language modeling.

Related papers

• One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks [26.848664285007022]
  ByT5-Sanskrit is designed for NLP applications involving the morphologically rich language Sanskrit. It is easier to deploy and more robust to data not covered by external linguistic resources. We show that our approach yields new best scores for lemmatization and dependency parsing of other morphologically rich languages.
  arXiv  Detail & Related papers  (2024-09-20T22:02:26Z)
• Walia-LLM: Enhancing Amharic-LLaMA by Integrating Task-Specific and Generative Datasets [2.8123257987021058]
  We focus on enhancing the LLaMA-2-Amharic model by integrating task-specific and generative datasets. We compile an Amharic instruction fine-tuning dataset and fine-tuned LLaMA-2-Amharic model. The fine-tuned model shows promising results in different NLP tasks.
  arXiv  Detail & Related papers  (2024-02-12T19:25:11Z)
• Paramanu: A Family of Novel Efficient Generative Foundation Language Models for Indian Languages [3.9018931027384056]
  We present "Paramanu", a family of novel language models (LM) for Indian languages. It covers 10 languages (Assamese, Bangla, Hindi, Konkani, Maithili, Marathi, Odia, Sanskrit, Tamil, Telugu) across 5 scripts. The models are pretrained on a single GPU with context size of 1024 and vary in size from 13.29 million (M) to 367.5 M parameters.
  arXiv  Detail & Related papers  (2024-01-31T17:58:10Z)
• YAYI 2: Multilingual Open-Source Large Language Models [53.92832054643197]
  We propose YAYI 2, including both base and chat models, with 30 billion parameters. YAYI 2 is pre-trained from scratch on a multilingual corpus which contains 2.65 trillion tokens filtered by our pre-training data processing pipeline. The base model is aligned with human values through supervised fine-tuning with millions of instructions and reinforcement learning from human feedback.
  arXiv  Detail & Related papers  (2023-12-22T17:34:47Z)
• Conversations in Galician: a Large Language Model for an Underrepresented Language [2.433983268807517]
  This paper introduces two novel resources designed to enhance Natural Language Processing (NLP) for the Galician language. We present a Galician adaptation of the Alpaca dataset, comprising 52,000 instructions and demonstrations. As a demonstration of the dataset utility, we fine-tuned LLaMA-7B to comprehend and respond in Galician, a language not originally supported by the model.
  arXiv  Detail & Related papers  (2023-11-07T08:52:28Z)
• FinGPT: Large Generative Models for a Small Language [48.46240937758779]
  We create large language models (LLMs) for Finnish, a language spoken by less than 0.1% of the world population. We train seven monolingual models from scratch (186M to 13B parameters) dubbed FinGPT. We continue the pretraining of the multilingual BLOOM model on a mix of its original training data and Finnish, resulting in a 176 billion parameter model we call BLUUMI.
  arXiv  Detail & Related papers  (2023-11-03T08:05:04Z)
• PolyLM: An Open Source Polyglot Large Language Model [57.64420154135178]
  We present PolyLM, a multilingual large language model (LLMs) trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning.
  arXiv  Detail & Related papers  (2023-07-12T09:00:37Z)
• Generalizing Multimodal Pre-training into Multilingual via Language Acquisition [54.69707237195554]
  English-based Vision-Language Pre-training has achieved great success in various downstream tasks. Some efforts have been taken to generalize this success to non-English languages through Multilingual Vision-Language Pre-training. We propose a textbfMultitextbfLingual textbfAcquisition (MLA) framework that can easily generalize a monolingual Vision-Language Pre-training model into multilingual.
  arXiv  Detail & Related papers  (2022-05-29T08:53:22Z)
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
  Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages. We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
  arXiv  Detail & Related papers  (2020-12-31T11:37:28Z)
• Mixed-Lingual Pre-training for Cross-lingual Summarization [54.4823498438831]
  Cross-lingual Summarization aims at producing a summary in the target language for an article in the source language. We propose a solution based on mixed-lingual pre-training that leverages both cross-lingual tasks like translation and monolingual tasks like masked language models. Our model achieves an improvement of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results.
  arXiv  Detail & Related papers  (2020-10-18T00:21:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.