Amortized Noisy Channel Neural Machine Translation

• URL: http://arxiv.org/abs/2112.08670v1
• Date: Thu, 16 Dec 2021 07:10:02 GMT
• Title: Amortized Noisy Channel Neural Machine Translation
• Abstract summary: Noisy channel models have been especially effective in neural machine translation (NMT) We aim to build an amortized noisy channel NMT model such that greedily decoding from it would generate translations that maximize the same reward as translations generated using BSR.
• Score: 53.48804610779759
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Noisy channel models have been especially effective in neural machine translation (NMT). However, recent approaches like "beam search and rerank" (BSR) incur significant computation overhead during inference, making real-world application infeasible. We aim to build an amortized noisy channel NMT model such that greedily decoding from it would generate translations that maximize the same reward as translations generated using BSR. We attempt three approaches: knowledge distillation, 1-step-deviation imitation learning, and Q learning. The first approach obtains the noisy channel signal from a pseudo-corpus, and the latter two approaches aim to optimize toward a noisy-channel MT reward directly. All three approaches speed up inference by 1-2 orders of magnitude. For all three approaches, the generated translations fail to achieve rewards comparable to BSR, but the translation quality approximated by BLEU is similar to the quality of BSR-produced translations.

Related papers

• DiffusionGemma Technical Report [72.10570774996219]
  We introduce DiffusionGemma, a language model that uses discrete diffusion to generate text at exceptionally high speed.<n>Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel.<n>Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget.
  arXiv  Detail & Related papers  (2026-07-31T16:11:46Z)
• Compensating for Data with Reasoning: Low-Resource Machine Translation with LLMs [0.0]
  Fragment-Shot Prompting is a novel in-context learning method that segments input and retrieves translation examples based on syntactic coverage.<n> Pivoted Fragment-Shot is an extension that enables translation without direct parallel data.<n>We evaluate these methods using GPT-3.5, GPT-4o, o1-mini, LLaMA-3.3, and DeepSeek-R1 for translation between Italian and two Ladin variants.
  arXiv  Detail & Related papers  (2025-05-28T12:29:05Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Improving Cascaded Unsupervised Speech Translation with Denoising Back-translation [70.33052952571884]
  We propose to build a cascaded speech translation system without leveraging any kind of paired data. We use fully unpaired data to train our unsupervised systems and evaluate our results on CoVoST 2 and CVSS.
  arXiv  Detail & Related papers  (2023-05-12T13:07:51Z)
• M3ST: Mix at Three Levels for Speech Translation [66.71994367650461]
  We propose Mix at three levels for Speech Translation (M3ST) method to increase the diversity of the augmented training corpus. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. Experiments on MuST-C speech translation benchmark and analysis show that M3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.
  arXiv  Detail & Related papers  (2022-12-07T14:22:00Z)
• An Empirical Study of Language Model Integration for Transducer based Speech Recognition [23.759084092602517]
  Methods such as density ratio (DR) and ILM estimation (ILME) have been developed, outperforming the classic shallow fusion (SF) method. We propose a low-order density ratio method (LODR) by training a low-order weak ILM for DR.
  arXiv  Detail & Related papers  (2022-03-31T03:33:50Z)
• Phrase-level Adversarial Example Generation for Neural Machine Translation [75.01476479100569]
  We propose a phrase-level adversarial example generation (PAEG) method to enhance the robustness of the model. We verify our method on three benchmarks, including LDC Chinese-English, IWSLT14 German-English, and WMT14 English-German tasks.
  arXiv  Detail & Related papers  (2022-01-06T11:00:49Z)
• Unsupervised Neural Machine Translation with Generative Language Models Only [19.74865387759671]
  We show how to derive state-of-the-art unsupervised neural machine translation systems from generatively pre-trained language models. Our method consists of three steps: few-shot amplification, distillation, and backtranslation.
  arXiv  Detail & Related papers  (2021-10-11T17:35:34Z)
• Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation [98.11249019844281]
  Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation (NAT) models. We propose reverse KD to rejuvenate more alignments for low-frequency target words. Results demonstrate that the proposed approach can significantly and universally improve translation quality.
  arXiv  Detail & Related papers  (2021-06-02T02:41:40Z)
• Addressing the Vulnerability of NMT in Input Perturbations [10.103375853643547]
  We improve robustness of NMT models by reducing the effect of noisy words through a Context-Enhanced Reconstruction approach. CER trains the model to resist noise in two steps: (1) step that breaks the naturalness of input sequence with made-up words; (2) reconstruction step that defends the noise propagation by generating better and more robust contextual representation.
  arXiv  Detail & Related papers  (2021-04-20T07:52:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.