On Evaluation of Bangla Word Analogies

• URL: http://arxiv.org/abs/2304.04613v1
• Date: Mon, 10 Apr 2023 14:27:35 GMT
• Title: On Evaluation of Bangla Word Analogies
• Authors: Mousumi Akter, Souvika Sarkar, Shubhra Kanti Karmaker (Santu)
• Abstract summary: This paper presents a high-quality dataset for evaluating the quality of Bangla word embeddings. Despite being the 7th most-spoken language in the world, Bangla is a low-resource language and popular NLP models fail to perform well.
• Score: 0.8658596218544772
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This paper presents a high-quality dataset for evaluating the quality of Bangla word embeddings, which is a fundamental task in the field of Natural Language Processing (NLP). Despite being the 7th most-spoken language in the world, Bangla is a low-resource language and popular NLP models fail to perform well. Developing a reliable evaluation test set for Bangla word embeddings are crucial for benchmarking and guiding future research. We provide a Mikolov-style word analogy evaluation set specifically for Bangla, with a sample size of 16678, as well as a translated and curated version of the Mikolov dataset, which contains 10594 samples for cross-lingual research. Our experiments with different state-of-the-art embedding models reveal that Bangla has its own unique characteristics, and current embeddings for Bangla still struggle to achieve high accuracy on both datasets. We suggest that future research should focus on training models with larger datasets and considering the unique morphological characteristics of Bangla. This study represents the first step towards building a reliable NLP system for the Bangla language1.

Related papers

• A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus [71.77214818319054]
  Natural language inference is a proxy for natural language understanding. There is no publicly available NLI corpus for the Romanian language. We introduce the first Romanian NLI corpus (RoNLI) comprising 58K training sentence pairs.
  arXiv  Detail & Related papers  (2024-05-20T08:41:15Z)
• Natural Language Processing for Dialects of a Language: A Survey [56.93337350526933]
  State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language. Motivated by the performance degradation of NLP models for dialectic datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches.
  arXiv  Detail & Related papers  (2024-01-11T03:04:38Z)
• BanglaNLP at BLP-2023 Task 2: Benchmarking different Transformer Models for Sentiment Analysis of Bangla Social Media Posts [0.46040036610482665]
  This paper presents our submission to Task 2 (Sentiment Analysis of Bangla Social Media Posts) of the BLP Workshop. Our quantitative results show that transfer learning really helps in better learning of the models in this low-resource language scenario. We obtain a micro-F1 of 67.02% on the test set and our performance in this shared task is ranked at 21 in the leaderboard.
  arXiv  Detail & Related papers  (2023-10-13T16:46:38Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• Ensemble Transfer Learning for Multilingual Coreference Resolution [60.409789753164944]
  A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. We design a simple but effective ensemble-based framework that combines various transfer learning techniques. We also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.
  arXiv  Detail & Related papers  (2023-01-22T18:22:55Z)
• BanglaParaphrase: A High-Quality Bangla Paraphrase Dataset [3.922582192616519]
  We present BanglaParaphrase, a high-quality synthetic Bangla Paraphrase dataset curated by a novel filtering pipeline. We aim to take a step towards alleviating the low resource status of the Bangla language in the NLP domain through the introduction of BanglaParaphrase.
  arXiv  Detail & Related papers  (2022-10-11T02:52:31Z)
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages [87.5457337866383]
  We introduce the Image-Grounded Language Understanding Evaluation benchmark. IGLUE brings together visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. We find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks.
  arXiv  Detail & Related papers  (2022-01-27T18:53:22Z)
• End-to-End Natural Language Understanding Pipeline for Bangla Conversational Agents [0.43012765978447565]
  We propose a novel approach to build a business assistant which can communicate in Bangla and Bangla Transliteration in English with high confidence consistently. We use Rasa Open Source Framework, fastText embeddings, Polyglot embeddings, Flask, and other systems as building blocks. We present a pipeline for intent classification and entity extraction which achieves reasonable performance.
  arXiv  Detail & Related papers  (2021-07-12T16:09:22Z)
• A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models [2.5768647103950357]
  We provide a review of Bangla NLP tasks, resources, and tools available to the research community. We benchmark datasets collected from various platforms for nine NLP tasks using current state-of-the-art algorithms. We report our results using both individual and consolidated datasets and provide data for future research.
  arXiv  Detail & Related papers  (2021-07-08T13:49:46Z)
• BanglaBERT: Combating Embedding Barrier for Low-Resource Language Understanding [1.7000879291900044]
  We build a Bangla natural language understanding model pre-trained on 18.6 GB data we crawled from top Bangla sites on the internet. Our model outperforms multilingual baselines and previous state-of-the-art results by 1-6%. We identify a major shortcoming of multilingual models that hurt performance for low-resource languages that don't share writing scripts with any high resource one.
  arXiv  Detail & Related papers  (2021-01-01T09:28:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.