Can the Variation of Model Weights be used as a Criterion for Self-Paced Multilingual NMT?

• URL: http://arxiv.org/abs/2410.04147v1
• Date: Sat, 5 Oct 2024 12:52:51 GMT
• Title: Can the Variation of Model Weights be used as a Criterion for Self-Paced Multilingual NMT?
• Authors: Àlex R. Atrio, Alexis Allemann, Ljiljana Dolamic, Andrei Popescu-Belis,
• Abstract summary: Many-to-one neural machine translation systems improve over one-to-one systems when training data is scarce. In this paper, we design and test a novel algorithm for selecting the language of minibatches when training such systems.
• Score: 7.330978520551704
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Many-to-one neural machine translation systems improve over one-to-one systems when training data is scarce. In this paper, we design and test a novel algorithm for selecting the language of minibatches when training such systems. The algorithm changes the language of the minibatch when the weights of the model do not evolve significantly, as measured by the smoothed KL divergence between all layers of the Transformer network. This algorithm outperforms the use of alternating monolingual batches, but not the use of shuffled batches, in terms of translation quality (measured with BLEU and COMET) and convergence speed.

Related papers

• Using Machine Translation to Augment Multilingual Classification [0.0]
  We explore the effects of using machine translation to fine-tune a multilingual model for a classification task across multiple languages. We show that translated data are of sufficient quality to tune multilingual classifiers and that this novel loss technique is able to offer some improvement over models tuned without it.
  arXiv  Detail & Related papers  (2024-05-09T00:31:59Z)
• Revisiting Machine Translation for Cross-lingual Classification [91.43729067874503]
  Most research in the area focuses on the multilingual models rather than the Machine Translation component. We show that, by using a stronger MT system and mitigating the mismatch between training on original text and running inference on machine translated text, translate-test can do substantially better than previously assumed.
  arXiv  Detail & Related papers  (2023-05-23T16:56:10Z)
• How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training? [86.48323488619629]
  We analyze how translation performance changes as the data ratios among languages vary in the tokenizer training corpus. We find that while relatively better performance is often observed when languages are more equally sampled, the downstream performance is more robust to language imbalance than we usually expected.
  arXiv  Detail & Related papers  (2022-04-29T17:50:36Z)
• Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval [66.69799641522133]
  State-of-the-art neural (re)rankers are notoriously data hungry. Current approaches typically transfer rankers trained on English data to other languages and cross-lingual setups by means of multilingual encoders. We show that two parameter-efficient approaches to cross-lingual transfer, namely Sparse Fine-Tuning Masks (SFTMs) and Adapters, allow for a more lightweight and more effective zero-shot transfer.
  arXiv  Detail & Related papers  (2022-04-05T15:44:27Z)
• Learning Kernel-Smoothed Machine Translation with Retrieved Examples [30.17061384497846]
  Existing non-parametric approaches that retrieve similar examples from a database to guide the translation process are promising but are prone to overfit the retrieved examples. We propose to learn Kernel-Smoothed Translation with Example Retrieval (KSTER), an effective approach to adapt neural machine translation models online.
  arXiv  Detail & Related papers  (2021-09-21T06:42:53Z)
• Distributionally Robust Multilingual Machine Translation [94.51866646879337]
  We propose a new learning objective for Multilingual neural machine translation (MNMT) based on distributionally robust optimization. We show how to practically optimize this objective for large translation corpora using an iterated best response scheme. Our method consistently outperforms strong baseline methods in terms of average and per-language performance under both many-to-one and one-to-many translation settings.
  arXiv  Detail & Related papers  (2021-09-09T03:48:35Z)
• Modelling Latent Translations for Cross-Lingual Transfer [47.61502999819699]
  We propose a new technique that integrates both steps of the traditional pipeline (translation and classification) into a single model. We evaluate our novel latent translation-based model on a series of multilingual NLU tasks. We report gains for both zero-shot and few-shot learning setups, up to 2.7 accuracy points on average.
  arXiv  Detail & Related papers  (2021-07-23T17:11:27Z)
• Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation [38.83163343372786]
  We propose a novel bilingual mutual information (BMI) based adaptive objective, which measures the learning difficulty for each target token from the perspective of bilingualism. Experimental results on WMT14 English-to-German and WMT19 Chinese-to-English demonstrate the superiority of our approach compared with the Transformer baseline and previous token-level adaptive training approaches.
  arXiv  Detail & Related papers  (2021-05-26T12:54:24Z)
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information [72.2412707779571]
  mRASP is an approach to pre-train a universal multilingual neural machine translation model. We carry out experiments on 42 translation directions across a diverse setting, including low, medium, rich resource, and as well as transferring to exotic language pairs.
  arXiv  Detail & Related papers  (2020-10-07T03:57:54Z)
• Deep Transformers with Latent Depth [42.33955275626127]
  The Transformer model has achieved state-of-the-art performance in many sequence modeling tasks. We present a probabilistic framework to automatically learn which layer(s) to use by learning the posterior distributions of layer selection. We propose a novel method to train one shared Transformer network for multilingual machine translation.
  arXiv  Detail & Related papers  (2020-09-28T07:13:23Z)
• It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information [90.35685796083563]
  Cross-mutual information (XMI) is an asymmetric information-theoretic metric of machine translation difficulty. XMI exploits the probabilistic nature of most neural machine translation models. We present the first systematic and controlled study of cross-lingual translation difficulties using modern neural translation systems.
  arXiv  Detail & Related papers  (2020-05-05T17:38:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.