Building a Large Japanese Web Corpus for Large Language Models

• URL: http://arxiv.org/abs/2404.17733v1
• Date: Sat, 27 Apr 2024 00:02:45 GMT
• Title: Building a Large Japanese Web Corpus for Large Language Models
• Abstract summary: This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive. This corpus consists of approximately 312.1 billion characters (approximately 173 million pages) To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs.
• Score: 20.407125127485276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.10 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora.

Related papers

• Multilingual Multi-Label Emotion Classification at Scale with Synthetic Data [2.3424047967193826]
  We construct a large-scale synthetic training corpus of over 1M multi-label samples across 23 languages.<n>We train and compare six multilingual transformer encoders, from DistilBERT (135M parameters) to XLM-R-Large (560M parameters)<n>To validate against human-annotated data, we evaluate all models zero-shot on GoEmotions (English) and SemEval-2018 Task 1 E-c (English, Arabic, Spanish)
  arXiv  Detail & Related papers  (2026-04-14T12:04:17Z)
• Towards High-Quality Machine Translation for Kokborok: A Low-Resource Tibeto-Burman Language of Northeast India [0.0]
  KokborokMT is a high-quality neural machine translation (NMT) system for Kokborok (ISO 639-3), a Tibeto-Burman language spoken primarily in Tripura, India.<n>We fine-tune the NLLB-200-distilled-600M model on a multi-source parallel corpus comprising 36,052 sentence pairs.<n>Our best system achieves BLEU scores of 17.30 and 38.56 on held-out test sets, representing substantial improvements over prior published results.
  arXiv  Detail & Related papers  (2026-03-28T18:20:13Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development [0.0]
  PashtoCorp is a 1.25-billion-word corpus for Pashto, a language spoken by 60 million people that remains underrepresented in NLP.<n>At 1.25B words across 2.81 million documents, PashtoCorp is 40x larger than the OSCAR Pashto subset and 83x larger than the previously largest dedicated Pashto corpus.
  arXiv  Detail & Related papers  (2026-03-17T10:36:18Z)
• Omnilingual MT: Machine Translation for 1,600 Languages [58.66170104105936]
  We present Omnilingual Machine Translation (OMT), the first machine translation system supporting more than 1,600 languages.<n>This scale is enabled by a comprehensive data strategy that integrates large public multilingual corpora with newly created datasets.<n>OMT models improve in cross-lingual transfer, being close to solving the "understanding" part of the puzzle in MT for the 1,600 evaluated.
  arXiv  Detail & Related papers  (2026-03-17T09:43:42Z)
• DIETA: A Decoder-only transformer-based model for Italian-English machine TrAnslation [74.85762984118024]
  DIETA is a small, decoder-only Transformer model with 0.5 billion parameters.<n>We collect and curate a large parallel corpus consisting of approximately 207 million Italian-English sentence pairs.<n>We release a new small-scale evaluation set, consisting of 450 sentences, based on 2025 WikiNews articles.
  arXiv  Detail & Related papers  (2026-01-25T13:08:43Z)
• Trillion 7B Technical Report [9.289348528268837]
  Trillion-7B is the most token-efficient Korean-centric multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA) mechanism enables highly efficient knowledge transfer from English to target languages like Korean and Japanese.
  arXiv  Detail & Related papers  (2025-04-21T20:54:44Z)
• Towards Robust Speech Representation Learning for Thousands of Languages [77.2890285555615]
  Self-supervised learning (SSL) has helped extend speech technologies to more languages by reducing the need for labeled data. We propose XEUS, a Cross-lingual for Universal Speech, trained on over 1 million hours of data across 4057 languages.
  arXiv  Detail & Related papers  (2024-06-30T21:40:26Z)
• DataComp-LM: In search of the next generation of training sets for language models [200.5293181577585]
  DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments with the goal of improving language models. We provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters.
  arXiv  Detail & Related papers  (2024-06-17T17:42:57Z)
• mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus [52.83121058429025]
  We introduce mOSCAR, the first large-scale multilingual and multimodal document corpus crawled from the web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. It shows a strong boost in few-shot learning performance across various multilingual image-text tasks and benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T00:13:32Z)
• A Japanese-Chinese Parallel Corpus Using Crowdsourcing for Web Mining [20.18032411452028]
  We created a Japanese-Chinese parallel corpus of 4.6M sentence pairs from bilingual websites. We used a Japanese-Chinese bilingual dictionary of 160K word pairs for document and sentence alignment. We compared the translation accuracy of the model trained on these 4.6M sentence pairs with that of the model trained on Japanese-Chinese sentence pairs from CCMatrix (12.4M), a parallel corpus from global web mining.
  arXiv  Detail & Related papers  (2024-05-15T00:54:40Z)
• Assessing Translation capabilities of Large Language Models involving English and Indian Languages [4.067706269490143]
  We explore the multilingual capabilities of large language models by using machine translation as a task involving English and 22 Indian languages. We fine-tune these large language models using parameter efficient fine-tuning methods such as LoRA and additionally with full fine-tuning. Our results demonstrate significant progress, with average BLEU scores of 13.42, 15.93, 12.13, 12.30, and 12.07, as well as CHRF scores of 43.98, 46.99, 42.55, 42.42, and 45.39, respectively.
  arXiv  Detail & Related papers  (2023-11-15T18:58:19Z)
• Skywork: A More Open Bilingual Foundation Model [55.927396986873816]
  We present Skywork-13B, a family of large language models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both English and Chinese texts. We show that our model not only excels on popular benchmarks, but also achieves emphstate of the art performance in Chinese language modeling on diverse domains.
  arXiv  Detail & Related papers  (2023-10-30T08:31:47Z)
• Baichuan 2: Open Large-scale Language Models [51.56361715162972]
  We present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval.
  arXiv  Detail & Related papers  (2023-09-19T04:13:22Z)
• Vega-MT: The JD Explore Academy Translation System for WMT22 [67.71016343958556]
  We push the limit of our previous work by scaling up two main factors, i.e. language pairs and model sizes. We get the 1st place on Zh-En (45.1), En-Zh (61.7), De-En (58.0), En-De (63.2), Cs-En (74.7), Ru-En (64.9), En-Ru (69.6), and En-Ja (65.1), 2nd place on En-Cs (95.3) and Ja-En (40.6), respectively.
  arXiv  Detail & Related papers  (2022-09-20T03:45:24Z)
• JParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpus [31.203776611871863]
  This paper creates a large parallel corpus for English-Japanese, a language pair for which only limited resources are available. It introduces a new web-based English-Japanese parallel corpus named JParaCrawl v3.0. Our new corpus contains more than 21 million unique parallel sentence pairs, which is more than twice as many as the previous JParaCrawl v2.0 corpus.
  arXiv  Detail & Related papers  (2022-02-25T10:52:00Z)
• CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model [15.469228003507919]
  We introduce the Chinese corpus from CLUE organization, CLUECorpus 2020. It has 100G raw corpus with 35 billion Chinese characters, which is retrieved from Common Crawl. We release a new Chinese vocabulary with a size of 8K, which is only one-third of the vocabulary size used in Chinese Bert released by Google.
  arXiv  Detail & Related papers  (2020-03-03T06:39:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.