L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models

• URL: http://arxiv.org/abs/2204.08398v1
• Date: Mon, 18 Apr 2022 16:49:59 GMT
• Title: L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models
• Authors: Ravindra Nayak, Raviraj Joshi
• Abstract summary: We present L3Cube-HingCorpus, the first large-scale real Hindi-English code mixed data in a Roman script. We show the effectiveness of these BERT models on the subsequent downstream tasks like code-mixed sentiment analysis, POS tagging, NER, and LID from the GLUECoS benchmark.
• Score: 1.14219428942199
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Code-switching occurs when more than one language is mixed in a given sentence or a conversation. This phenomenon is more prominent on social media platforms and its adoption is increasing over time. Therefore code-mixed NLP has been extensively studied in the literature. As pre-trained transformer-based architectures are gaining popularity, we observe that real code-mixing data are scarce to pre-train large language models. We present L3Cube-HingCorpus, the first large-scale real Hindi-English code mixed data in a Roman script. It consists of 52.93M sentences and 1.04B tokens, scraped from Twitter. We further present HingBERT, HingMBERT, HingRoBERTa, and HingGPT. The BERT models have been pre-trained on codemixed HingCorpus using masked language modelling objectives. We show the effectiveness of these BERT models on the subsequent downstream tasks like code-mixed sentiment analysis, POS tagging, NER, and LID from the GLUECoS benchmark. The HingGPT is a GPT2 based generative transformer model capable of generating full tweets. We also release L3Cube-HingLID Corpus, the largest code-mixed Hindi-English language identification(LID) dataset and HingBERT-LID, a production-quality LID model to facilitate capturing of more code-mixed data using the process outlined in this work. The dataset and models are available at https://github.com/l3cube-pune/code-mixed-nlp .

Related papers

• TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data [50.40191599304911]
  We propose Transliterate transliteration-Merge (TransMI), which can create a strong baseline well-suited for data that is transliterated into a common script. Results show a consistent improvement of 3% to 34%, varying across different models and tasks.
  arXiv  Detail & Related papers  (2024-05-16T09:08:09Z)
• Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindi [0.0]
  We introduce Tri-Distil-BERT, a multilingual model pre-trained on Bangla, English, and Hindi, and Mixed-Distil-BERT, a model fine-tuned on code-mixed data. Our two-tiered pre-training approach offers efficient alternatives for multilingual and code-mixed language understanding.
  arXiv  Detail & Related papers  (2023-09-19T02:59:41Z)
• My Boli: Code-mixed Marathi-English Corpora, Pretrained Language Models and Evaluation Benchmarks [0.7874708385247353]
  We focus on the low-resource Indian language Marathi which lacks any prior work in code-mixing. We present L3Cube-MeCorpus, a large code-mixed Marathi-English (Mr-En) corpus with 10 million social media sentences for pretraining. We also release L3Cube-MeBERT and MeRoBERTa, code-mixed BERT-based transformer models pre-trained on MeCorpus.
  arXiv  Detail & Related papers  (2023-06-24T18:17:38Z)
• Leveraging Language Identification to Enhance Code-Mixed Text Classification [0.7340017786387767]
  Existing deep-learning models do not take advantage of the implicit language information in code-mixed text. Our study aims to improve BERT-based models performance on low-resource Code-Mixed Hindi-English datasets.
  arXiv  Detail & Related papers  (2023-06-08T06:43:10Z)
• Comparative Study of Pre-Trained BERT Models for Code-Mixed Hindi-English Data [0.7874708385247353]
  "Code Mixed" refers to the use of more than one language in the same text. In this work, we focus on low-resource Hindi-English code-mixed language. We report state-of-the-art results on respective datasets using HingBERT-based models.
  arXiv  Detail & Related papers  (2023-05-25T05:10:28Z)
• Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages [47.78634360870564]
  We explore prompting multilingual models to generate code-mixed data for seven languages in South East Asia (SEA) We find that publicly available multilingual instruction-tuned models such as BLOOMZ are incapable of producing texts with phrases or clauses from different languages. ChatGPT exhibits inconsistent capabilities in generating code-mixed texts, wherein its performance varies depending on the prompt template and language pairing.
  arXiv  Detail & Related papers  (2023-03-23T18:16:30Z)
• Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval [87.11836738011007]
  We propose a multilingual multilingual language model called masked sentence model (MSM) MSM consists of a sentence encoder to generate the sentence representations, and a document encoder applied to a sequence of sentence vectors from a document. To train the model, we propose a masked sentence prediction task, which masks and predicts the sentence vector via a hierarchical contrastive loss with sampled negatives.
  arXiv  Detail & Related papers  (2023-02-03T09:54:27Z)
• Transformer-based Model for Word Level Language Identification in Code-mixed Kannada-English Texts [55.41644538483948]
  We propose the use of a Transformer based model for word-level language identification in code-mixed Kannada English texts. The proposed model on the CoLI-Kenglish dataset achieves a weighted F1-score of 0.84 and a macro F1-score of 0.61.
  arXiv  Detail & Related papers  (2022-11-26T02:39:19Z)
• Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing [19.19256927651015]
  We describe models that convert monolingual English text into Hinglish (code-mixed Hindi and English) Given the recent success of pretrained language models, we also test the utility of two recent Transformer-based encoder-decoder models. Our models place first in the overall ranking of the English-Hinglish official shared task.
  arXiv  Detail & Related papers  (2021-05-18T19:50:25Z)
• Paraphrastic Representations at Scale [134.41025103489224]
  We release trained models for English, Arabic, German, French, Spanish, Russian, Turkish, and Chinese languages. We train these models on large amounts of data, achieving significantly improved performance from the original papers.
  arXiv  Detail & Related papers  (2021-04-30T16:55:28Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.