Announcing CzEng 2.0 Parallel Corpus with over 2 Gigawords

• URL: http://arxiv.org/abs/2007.03006v1
• Date: Mon, 6 Jul 2020 18:48:11 GMT
• Title: Announcing CzEng 2.0 Parallel Corpus with over 2 Gigawords
• Abstract summary: We present a new release of the Czech-English parallel corpus CzEng 2.0 consisting of over 2 billion words (2 "gigawords") in each language. The corpus contains document-level information and is filtered with several techniques to lower the amount of noise.
• Score: 12.561914014993675
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a new release of the Czech-English parallel corpus CzEng 2.0 consisting of over 2 billion words (2 "gigawords") in each language. The corpus contains document-level information and is filtered with several techniques to lower the amount of noise. In addition to the data in the previous version of CzEng, it contains new authentic and also high-quality synthetic parallel data. CzEng is freely available for research and educational purposes.

Related papers

• SiDiaC-v.2.0: Sinhala Diachronic Corpus Version 2.0 [0.9001507266214666]
  SiDiaC-v.2.0 is the largest comprehensive Sinhala Diachronic to date, covering a period from 1800 CE to 1955 CE.<n>The corpus consists of 244k words across 185 literary works that underwent thorough filtering, preprocessing, and copyright compliance checks.<n>A subset of 59 documents totalling 70k words was annotated based on their written dates.
  arXiv  Detail & Related papers  (2026-03-11T15:10:32Z)
• SinhaLegal: A Benchmark Corpus for Information Extraction and Analysis in Sinhala Legislative Texts [0.0]
  SinhaLegal introduces a Sinhala legislative text corpus containing approximately 2 million words across 1,206 legal documents.<n>The dataset includes two types of legal documents: 1,065 Acts dated from 1981 to 2014 and 141 Bills from 2010 to 2014.<n>The texts were extracted using OCR with Google Document AI, followed by extensive post-processing and manual cleaning to ensure high-quality, machine-readable content.
  arXiv  Detail & Related papers  (2026-03-05T06:13:44Z)
• NeuCLIRTech: Chinese Monolingual and Cross-Language Information Retrieval Evaluation in a Challenging Domain [49.3943974580576]
  This paper presents NeuCLIRTech, an evaluation collection for cross-language retrieval over technical information.<n>The collection consists of technical documents written in Chinese and those same documents machine translated into English.<n>The collection supports two retrieval scenarios: monolingual retrieval in Chinese, and cross-language retrieval with English as the query language.
  arXiv  Detail & Related papers  (2026-02-05T05:57:55Z)
• 20min-XD: A Comparable Corpus of Swiss News Articles [42.49142747741821]
  We present 20min-XD (20 Minuten cross-lingual document-level), a French-German, document-level comparable corpus of news articles. Our dataset comprises around 15,000 article pairs spanning 2015 to 2024, automatically aligned based on semantic similarity. The resulting dataset exhibits a broad spectrum of cross-lingual similarity, ranging from near-translations to loosely related articles.
  arXiv  Detail & Related papers  (2025-04-30T14:16:08Z)
• Bilingual Corpus Mining and Multistage Fine-Tuning for Improving Machine Translation of Lecture Transcripts [50.00305136008848]
  We propose a framework for parallel corpus mining, which provides a quick and effective way to mine a parallel corpus from publicly available lectures on Coursera. For both English--Japanese and English--Chinese lecture translations, we extracted parallel corpora of approximately 50,000 lines and created development and test sets. This study also suggests guidelines for gathering and cleaning corpora, mining parallel sentences, cleaning noise in the mined data, and creating high-quality evaluation splits.
  arXiv  Detail & Related papers  (2023-11-07T03:50:25Z)
• SAHAAYAK 2023 -- the Multi Domain Bilingual Parallel Corpus of Sanskrit to Hindi for Machine Translation [0.0]
  The corpus contains total of 1.5M sentence pairs between Sanskrit and Hindi. Data from multiple domain has been incorporated into the corpus that includes, News, Daily conversations, Politics, History, Sport, and Ancient Indian Literature.
  arXiv  Detail & Related papers  (2023-06-27T11:06:44Z)
• Textless Speech-to-Speech Translation With Limited Parallel Data [51.3588490789084]
  PFB is a framework for training textless S2ST models that require just dozens of hours of parallel speech data. We train and evaluate our models for English-to-German, German-to-English and Marathi-to-English translation on three different domains.
  arXiv  Detail & Related papers  (2023-05-24T17:59:05Z)
• esCorpius: A Massive Spanish Crawling Corpus [2.262838186547612]
  esCorpius is a Spanish crawling corpus obtained from near 1 Pb of Common Crawl data. It is the most extensive corpus in Spanish with this level of quality in the extraction, purification and deduplication of web textual content.
  arXiv  Detail & Related papers  (2022-06-30T09:29:18Z)
• EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation [63.88541605363555]
  "Extract and Generate" (EAG) is a two-step approach to construct large-scale and high-quality multi-way aligned corpus from bilingual data. We first extract candidate aligned examples by pairing the bilingual examples from different language pairs with highly similar source or target sentences. We then generate the final aligned examples from the candidates with a well-trained generation model.
  arXiv  Detail & Related papers  (2022-03-04T08:21:27Z)
• JParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpus [31.203776611871863]
  This paper creates a large parallel corpus for English-Japanese, a language pair for which only limited resources are available. It introduces a new web-based English-Japanese parallel corpus named JParaCrawl v3.0. Our new corpus contains more than 21 million unique parallel sentence pairs, which is more than twice as many as the previous JParaCrawl v2.0 corpus.
  arXiv  Detail & Related papers  (2022-02-25T10:52:00Z)
• Models and Datasets for Cross-Lingual Summarisation [78.56238251185214]
  We present a cross-lingual summarisation corpus with long documents in a source language associated with multi-sentence summaries in a target language. The corpus covers twelve language pairs and directions for four European languages, namely Czech, English, French and German. We derive cross-lingual document-summary instances from Wikipedia by combining lead paragraphs and articles' bodies from language aligned Wikipedia titles.
  arXiv  Detail & Related papers  (2022-02-19T11:55:40Z)
• KazakhTTS2: Extending the Open-Source Kazakh TTS Corpus With More Data, Speakers, and Topics [4.859986264602551]
  We present an expanded version of our previously released Kazakh text-to-speech (KazakhTTS) synthesis corpus. In the new KazakhTTS2 corpus, the overall size is increased from 93 hours to 271 hours. The number of speakers has risen from two to five (three females and two males), and the topic coverage is diversified with the help of new sources, including a book and Wikipedia articles.
  arXiv  Detail & Related papers  (2022-01-15T06:54:30Z)
• ESPnet-ST IWSLT 2021 Offline Speech Translation System [56.83606198051871]
  This paper describes the ESPnet-ST group's IWSLT 2021 submission in the offline speech translation track. This year we made various efforts on training data, architecture, and audio segmentation. Our best E2E system combined all the techniques with model ensembling and achieved 31.4 BLEU.
  arXiv  Detail & Related papers  (2021-07-01T17:49:43Z)
• CoVoST 2 and Massively Multilingual Speech-to-Text Translation [24.904548615918355]
  CoVoST 2 is a large-scale multilingual speech translation corpus covering translations from 21 languages into English and from English into 15 languages. This represents the largest open dataset available to date from total volume and language coverage perspective.
  arXiv  Detail & Related papers  (2020-07-20T17:53:35Z)
• CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus [57.641761472372814]
  CoVoST is a multilingual speech-to-text translation corpus from 11 languages into English. It diversified with over 11,000 speakers and over 60 accents. CoVoST is released under CC0 license and free to use.
  arXiv  Detail & Related papers  (2020-02-04T14:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.