CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation

• URL: http://arxiv.org/abs/2506.19952v1
• Date: Tue, 24 Jun 2025 18:56:57 GMT
• Title: CycleDistill: Bootstrapping Machine Translation using LLMs with Cyclical Distillation
• Authors: Deepon Halder, Thanmay Jayakumar, Raj Dabre,
• Abstract summary: CycleDistill is a bootstrapping approach that generates synthetic parallel corpora from monolingual corpora via zero- or few-shot MT.<n>By relying solely on monolingual corpora, CycleDistill can achieve high-quality machine translation, improving upon a few-shot baseline model by over 20-30 chrF points in the first iteration.<n>We also study the effect of leveraging softmax activations during the distillation process and observe mild improvements in translation quality.
• Score: 10.892677818915255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs), despite their ability to perform few-shot machine translation (MT), often lag behind dedicated MT systems trained on parallel corpora, which are crucial for high quality machine translation (MT). However, parallel corpora are often scarce or non-existent for low-resource languages. In this paper, we propose CycleDistill, a bootstrapping approach leveraging LLMs and few-shot translation to obtain high-quality MT systems. CycleDistill involves iteratively generating synthetic parallel corpora from monolingual corpora via zero- or few-shot MT, which is then used to fine-tune the model that was used for generating said data for MT. CycleDistill does not need parallel corpora beyond 1 to 4 few-shot examples, and in our experiments focusing on three Indian languages, by relying solely on monolingual corpora, it can achieve high-quality machine translation, improving upon a few-shot baseline model by over 20-30 chrF points on average in the first iteration. We also study the effect of leveraging softmax activations during the distillation process and observe mild improvements in translation quality.

Related papers

• Trans-Zero: Self-Play Incentivizes Large Language Models for Multilingual Translation Without Parallel Data [64.4458540273004]
  We propose a self-play framework that leverages only monolingual data and the intrinsic multilingual knowledge of Large Language Models (LLMs)<n>Experiments demonstrate that this approach not only matches the performance of models trained on large-scale parallel data but also excels in non-English translation directions.
  arXiv  Detail & Related papers  (2025-04-20T16:20:30Z)
• Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation [62.202893186343935]
  We explore what it would take to adapt Large Language Models for low-resource languages. We show that parallel data is critical during both pre-training andSupervised Fine-Tuning (SFT) Our experiments with three LLMs across two low-resourced language groups reveal consistent trends, underscoring the generalizability of our findings.
  arXiv  Detail & Related papers  (2024-08-23T00:59:38Z)
• Machine Translation for Ge'ez Language [0.0]
  Machine translation for low-resource languages such as Ge'ez faces challenges such as out-of-vocabulary words, domain mismatches, and lack of labeled training data. We develop a multilingual neural machine translation (MNMT) model based on languages relatedness. We also experiment with using GPT-3.5, a state-of-the-art LLM, for few-shot translation with fuzzy matches.
  arXiv  Detail & Related papers  (2023-11-24T14:55:23Z)
• Boosting Unsupervised Machine Translation with Pseudo-Parallel Data [2.900810893770134]
  We propose a training strategy that relies on pseudo-parallel sentence pairs mined from monolingual corpora and synthetic sentence pairs back-translated from monolingual corpora. We reach an improvement of up to 14.5 BLEU points (English to Ukrainian) over a baseline trained on back-translated data only.
  arXiv  Detail & Related papers  (2023-10-22T10:57:12Z)
• Distilling Efficient Language-Specific Models for Cross-Lingual Transfer [75.32131584449786]
  Massively multilingual Transformers (MMTs) are widely used for cross-lingual transfer learning. MMTs' language coverage makes them unnecessarily expensive to deploy in terms of model size, inference time, energy, and hardware cost. We propose to extract compressed, language-specific models from MMTs which retain the capacity of the original MMTs for cross-lingual transfer.
  arXiv  Detail & Related papers  (2023-06-02T17:31:52Z)
• Revisiting Machine Translation for Cross-lingual Classification [91.43729067874503]
  Most research in the area focuses on the multilingual models rather than the Machine Translation component. We show that, by using a stronger MT system and mitigating the mismatch between training on original text and running inference on machine translated text, translate-test can do substantially better than previously assumed.
  arXiv  Detail & Related papers  (2023-05-23T16:56:10Z)
• Exploiting Language Relatedness in Machine Translation Through Domain Adaptation Techniques [3.257358540764261]
  We present a novel approach of using a scaled similarity score of sentences, especially for related languages based on a 5-gram KenLM language model. Our approach succeeds in increasing 2 BLEU point on multi-domain approach, 3 BLEU point on fine-tuning for NMT and 2 BLEU point on iterative back-translation approach.
  arXiv  Detail & Related papers  (2023-03-03T09:07:30Z)
• IsometricMT: Neural Machine Translation for Automatic Dubbing [9.605781943224251]
  This work introduces a self-learning approach that allows a transformer model to directly learn to generate outputs that closely match the source length. We report results on four language pairs with a publicly available benchmark based on TED Talk data.
  arXiv  Detail & Related papers  (2021-12-16T08:03:20Z)
• Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task [95.06453182273027]
  This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation. Our model submissions to the shared task were with DeltaLMnotefooturlhttps://aka.ms/deltalm, a generic pre-trained multilingual-decoder model. Our final submissions ranked first on three tracks in terms of the automatic evaluation metric.
  arXiv  Detail & Related papers  (2021-11-03T09:16:17Z)
• Efficient Inference for Multilingual Neural Machine Translation [60.10996883354372]
  We consider several ways to make multilingual NMT faster at inference without degrading its quality. Our experiments demonstrate that combining a shallow decoder with vocabulary filtering leads to more than twice faster inference with no loss in translation quality.
  arXiv  Detail & Related papers  (2021-09-14T13:28:13Z)
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information [72.2412707779571]
  mRASP is an approach to pre-train a universal multilingual neural machine translation model. We carry out experiments on 42 translation directions across a diverse setting, including low, medium, rich resource, and as well as transferring to exotic language pairs.
  arXiv  Detail & Related papers  (2020-10-07T03:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.