HindiLLM: Large Language Model for Hindi

• URL: http://arxiv.org/abs/2412.20357v1
• Date: Sun, 29 Dec 2024 05:28:15 GMT
• Title: HindiLLM: Large Language Model for Hindi
• Authors: Sanjay Chouhan, Shubha Brata Nath, Aparajita Dutta,
• Abstract summary: We have pre-trained two autoregressive Large Language Model (LLM) models for the Hindi language. We use a two-step process comprising unsupervised pre-training and supervised fine-tuning. The evaluation shows that the HindiLLM-based fine-tuned models outperform several models in most of the language related tasks.
• Score: 0.09363323206192666
• License:
• Abstract: The advancements in the Large Language Model (LLM) have helped in solving several problems related to language processing. Most of the researches have focused on the English language only, because of its popularity and abundance on the internet. However, a high-performance language model for Hindi and other Indic languages is lacking in the literature. In this work, we have pre-trained two autoregressive LLM models for the Hindi language, namely HindiLLM-Small and HindiLLM-Medium. We use a two-step process comprising unsupervised pre-training and supervised fine-tuning. First, we create a large and high-quality text corpus for unsupervised pre-training. Next, we train a Byte-Pair Encoding, named HindiLLM tokenizer, using the pre-training text data. We then perform training on the unlabeled data, known as the pre-training step, to get the HindiLLM base models. Furthermore, we perform fine-tuning of the HindiLLM base models for different tasks like sentiment analysis, text classification, natural language inference, and multiple choice question-answer on popular labeled datasets to measure the real-world performance. The evaluation shows that the HindiLLM-based fine-tuned models outperform several models in most of the language related tasks.

Related papers

• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Crosslingual Generalization through Multitask Finetuning [80.8822603322471]
  Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages.
  arXiv  Detail & Related papers  (2022-11-03T13:19:32Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Extractive Question Answering on Queries in Hindi and Tamil [2.66512000865131]
  Indic languages like Hindi and Tamil are underrepresented in the natural language processing (NLP) field compared to languages like English. The goal of this project is to build an NLP model that performs better than pre-existing models for the task of extractive question-answering (QA) on a public dataset in Hindi and Tamil.
  arXiv  Detail & Related papers  (2022-09-27T00:40:21Z)
• Multilingual Text Classification for Dravidian Languages [4.264592074410622]
  We propose a multilingual text classification framework for the Dravidian languages. On the one hand, the framework used the LaBSE pre-trained model as the base model. On the other hand, in view of the problem that the model cannot well recognize and utilize the correlation among languages, we further proposed a language-specific representation module.
  arXiv  Detail & Related papers  (2021-12-03T04:26:49Z)
• HinFlair: pre-trained contextual string embeddings for pos tagging and text classification in the Hindi language [0.0]
  HinFlair is a language representation model (contextual string embeddings) pre-trained on a large monolingual Hindi corpus. Results show that HinFlair outperforms previous state-of-the-art publicly available pre-trained embeddings for downstream tasks like text classification and pos tagging.
  arXiv  Detail & Related papers  (2021-01-18T09:23:35Z)
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
  Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages. We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
  arXiv  Detail & Related papers  (2020-12-31T11:37:28Z)
• Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages [0.8155575318208631]
  Language models based on the Transformer architecture have achieved state-of-the-art performance on a wide range of NLP tasks. However, this performance is usually tested and reported on high-resource languages, like English, French, Spanish, and German. Indian languages, on the other hand, are underrepresented in such benchmarks.
  arXiv  Detail & Related papers  (2020-11-04T14:43:43Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)
• WikiBERT models: deep transfer learning for many languages [1.3455090151301572]
  We introduce a simple, fully automated pipeline for creating languagespecific BERT models from Wikipedia data. We assess the merits of these models using the state-of-the-art UDify on Universal Dependencies data.
  arXiv  Detail & Related papers  (2020-06-02T11:57:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.