Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora

• URL: http://arxiv.org/abs/2402.07446v3
• Date: Fri, 14 Jun 2024 08:50:53 GMT
• Title: Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora
• Authors: Surangika Ranathunga, Nisansa de Silva, Menan Velayuthan, Aloka Fernando, Charitha Rathnayake,
• Abstract summary: We show that there are significant quality differences between different portions of web-mined corpora. We also show that, for some web-mined datasets, Neural Machine Translation (NMT) models trained with their highest-ranked 25k portion can be on par with human-curated datasets.
• Score: 1.0995326465245927
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We conducted a detailed analysis on the quality of web-mined corpora for two low-resource languages (making three language pairs, English-Sinhala, English-Tamil and Sinhala-Tamil). We ranked each corpus according to a similarity measure and carried out an intrinsic and extrinsic evaluation on different portions of this ranked corpus. We show that there are significant quality differences between different portions of web-mined corpora and that the quality varies across languages and datasets. We also show that, for some web-mined datasets, Neural Machine Translation (NMT) models trained with their highest-ranked 25k portion can be on par with human-curated datasets.

Related papers

• An Empirical Study on the Robustness of Massively Multilingual Neural Machine Translation [40.08063412966712]
  Massively multilingual neural machine translation (MMNMT) has been proven to enhance the translation quality of low-resource languages. We create a robustness evaluation benchmark dataset for Indonesian-Chinese translation. This dataset is automatically translated into Chinese using four NLLB-200 models of different sizes.
  arXiv  Detail & Related papers  (2024-05-13T12:01:54Z)
• Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages [11.512925610019474]
  We compare four of the most relevant large, web-crawled corpora across eleven lower-resourced European languages. We find that there are clear differences in quality of the corpora, with MaCoCu and OSCAR obtaining the best results. We conclude that, in our experiments, the quality of the web-crawled corpora does not seem to play a significant role when training LMs.
  arXiv  Detail & Related papers  (2024-03-13T16:56:33Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism [9.212370563134206]
  We show that content on the web is often translated into many languages, and the low quality of these multi-way translations indicates they were likely created using Machine Translation (MT) We also find evidence of a selection bias in the type of content which is translated into many languages, consistent with low quality English content being translated en masse into many lower resource languages, via MT. Our work raises serious concerns about training models such as multilingual large language models on both monolingual and bilingual data scraped from the web.
  arXiv  Detail & Related papers  (2024-01-11T08:56:13Z)
• What's In My Big Data? [67.04525616289949]
  We propose What's In My Big Data? (WIMBD), a platform and a set of sixteen analyses that allow us to reveal and compare the contents of large text corpora. WIMBD builds on two basic capabilities -- count and search -- at scale, which allows us to analyze more than 35 terabytes on a standard compute node. Our analysis uncovers several surprising and previously undocumented findings about these corpora, including the high prevalence of duplicate, synthetic, and low-quality content.
  arXiv  Detail & Related papers  (2023-10-31T17:59:38Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• Unified Model Learning for Various Neural Machine Translation [63.320005222549646]
  Existing machine translation (NMT) studies mainly focus on developing dataset-specific models. We propose a versatile'' model, i.e., the Unified Model Learning for NMT (UMLNMT) that works with data from different tasks. OurNMT results in substantial improvements over dataset-specific models with significantly reduced model deployment costs.
  arXiv  Detail & Related papers  (2023-05-04T12:21:52Z)
• FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation [64.9546787488337]
  We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese.
  arXiv  Detail & Related papers  (2022-10-01T05:02:04Z)
• DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages [5.367993194110256]
  DivEMT is the first publicly available post-editing study of Neural Machine Translation (NMT) over a typologically diverse set of target languages. We assess the impact on translation productivity of two state-of-the-art NMT systems, namely: Google Translate and the open-source multilingual model mBART50.
  arXiv  Detail & Related papers  (2022-05-24T17:22:52Z)
• ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback [70.5469946314539]
  ChrEnTranslate is an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability.
  arXiv  Detail & Related papers  (2021-07-30T17:58:54Z)
• Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets [21.375943264243144]
  We manually audit the quality of 205 language-specific corpora released with five major public datasets. We find that at least 15 corpora are completely erroneous, and a significant fraction contains less than 50% sentences of acceptable quality. We demonstrate that these issues are easy to detect even for non-speakers of the languages in question, and supplement the human judgements with automatic analyses.
  arXiv  Detail & Related papers  (2021-03-22T17:30:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.