Trillion 7B Technical Report

• URL: http://arxiv.org/abs/2504.15431v1
• Date: Mon, 21 Apr 2025 20:54:44 GMT
• Title: Trillion 7B Technical Report
• Abstract summary: Trillion-7B is the most token-efficient Korean-centric multilingual LLM available.<n>Our novel Cross-lingual Document Attention (XLDA) mechanism enables highly efficient knowledge transfer from English to target languages like Korean and Japanese.
• Score: 9.289348528268837
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Trillion-7B, the most token-efficient Korean-centric multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA) mechanism enables highly efficient and effective knowledge transfer from English to target languages like Korean and Japanese. Combined with optimized data mixtures, language-specific filtering, and tailored tokenizer construction, Trillion-7B achieves competitive performance while dedicating only 10\% of its 2T training tokens to multilingual data and requiring just 59.4K H100 GPU hours (\$148K) for full training. Comprehensive evaluations across 27 benchmarks in four languages demonstrate Trillion-7B's robust multilingual performance and exceptional cross-lingual consistency.

Related papers

• MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages [30.09756627209374]
  MultiSynt/MT is an open synthetic parallel corpus with approximately 4.8 trillion target- tokens across 36 European languages.<n>For many medium- and lower-resource European languages, this is the largest openly available pre-training resource.
  arXiv  Detail & Related papers  (2026-07-01T12:55:58Z)
• ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset [16.940142252787144]
  We study multilingual data curation across thirteen languages.<n>In controlled bilingual experiments, improving data quality for any single language benefits others.<n>We operationalize this approach within an effort that produced a 20T-token pretraining corpus.
  arXiv  Detail & Related papers  (2026-02-16T21:40:03Z)
• Comprehension of Multilingual Expressions Referring to Target Objects in Visual Inputs [47.944645462877894]
  Referring Expression (REC) requires models to localize objects in images based on natural language descriptions.<n>This work addresses multilingual REC through two main contributions.<n>We construct a unified multilingual dataset spanning 10 languages, by systematically expanding 12 existing English REC benchmarks through machine translation and context-based translation enhancement.<n>The resulting dataset comprises approximately 8 million multilingual referring expressions across 177,620 images, with 336,882 annotated objects.
  arXiv  Detail & Related papers  (2025-11-14T15:54:34Z)
• IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs [5.068673710249497]
  IndicSuperTokenizer is a tokenizer for Indic multilingual LLMs.<n>It combines subword and multi-word tokenization, along with language-specific tokens pre-tokenization.<n>It improves the average fertility score by 39.5% over LLaMA4 and by 18% over Sutra.
  arXiv  Detail & Related papers  (2025-11-05T06:57:42Z)
• fact check AI at SemEval-2025 Task 7: Multilingual and Crosslingual Fact-checked Claim Retrieval [0.0]
  Training used both the source languages and their English translations for multilingual retrieval and only English translations for cross-lingual retrieval.<n>The method achieved 92% Success@10 in multilingual and 80% Success@10 in 5th in crosslingual and 10th in multilingual tracks.
  arXiv  Detail & Related papers  (2025-08-05T14:10:09Z)
• One ruler to measure them all: Benchmarking multilingual long-context language models [35.75388430206553]
  We present ONERULER, a multilingual benchmark designed to evaluate long-context language models across 26 languages. English is not the top-performing language on long-context tasks (ranked 6th out of 26), with Polish emerging as the top language. In cross-lingual scenarios where instructions and context appear in different languages, performance can fluctuate by up to 20% depending on the instruction language.
  arXiv  Detail & Related papers  (2025-03-03T19:12:48Z)
• Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model [66.17354128553244]
  Most Large Vision-Language Models (LVLMs) to date are trained predominantly on English data.<n>We investigate how different training mixes tip the scale for different groups of languages.<n>We train Centurio, a 100-language LVLM, offering state-of-the-art performance in an evaluation covering 14 tasks and 56 languages.
  arXiv  Detail & Related papers  (2025-01-09T10:26:14Z)
• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations [59.056367787688146]
  This paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. We construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages. By utilizing translation, we construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages.
  arXiv  Detail & Related papers  (2023-10-31T08:09:20Z)
• KInITVeraAI at SemEval-2023 Task 3: Simple yet Powerful Multilingual Fine-Tuning for Persuasion Techniques Detection [0.0]
  This paper presents the best-performing solution to the SemEval 2023 Task 3 on the subtask 3 dedicated to persuasion techniques detection. Due to a high multilingual character of the input data and a large number of 23 predicted labels, we opted for fine-tuning pre-trained transformer-based language models.
  arXiv  Detail & Related papers  (2023-04-24T09:06:43Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation [105.41167108465085]
  Cross-lingual Machine Reading (CLMRC) remains a challenging problem due to the lack of large-scale datasets in low-source languages. We propose a novel augmentation approach named Language Branch Machine Reading (LBMRC) LBMRC trains multiple machine reading comprehension (MRC) models proficient in individual language. We devise a multilingual distillation approach to amalgamate knowledge from multiple language branch models to a single model for all target languages.
  arXiv  Detail & Related papers  (2020-10-27T13:12:17Z)
• Looking for Clues of Language in Multilingual BERT to Improve Cross-lingual Generalization [56.87201892585477]
  Token embeddings in multilingual BERT (m-BERT) contain both language and semantic information. We control the output languages of multilingual BERT by manipulating the token embeddings.
  arXiv  Detail & Related papers  (2020-10-20T05:41:35Z)
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT [60.9051207862378]
  multilingual BERT works remarkably well on cross-lingual transfer tasks. Datasize and context window size are crucial factors to the transferability. There is a computationally cheap but effective approach to improve the cross-lingual ability of multilingual BERT.
  arXiv  Detail & Related papers  (2020-04-20T11:13:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.