A Systematic Analysis of Vocabulary and BPE Settings for Optimal Fine-tuning of NMT: A Case Study of In-domain Translation

• URL: http://arxiv.org/abs/2303.00722v1
• Date: Wed, 1 Mar 2023 18:26:47 GMT
• Title: A Systematic Analysis of Vocabulary and BPE Settings for Optimal Fine-tuning of NMT: A Case Study of In-domain Translation
• Authors: J. Pourmostafa Roshan Sharami, D. Shterionov, P. Spronck
• Abstract summary: The choice of vocabulary and SW tokenization has a significant impact on both training and fine-tuning an NMT model. In this work we compare different strategies for SW tokenization and vocabulary generation with the ultimate goal to uncover an optimal setting for fine-tuning a domain-specific model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The effectiveness of Neural Machine Translation (NMT) models largely depends on the vocabulary used at training; small vocabularies can lead to out-of-vocabulary problems -- large ones, to memory issues. Subword (SW) tokenization has been successfully employed to mitigate these issues. The choice of vocabulary and SW tokenization has a significant impact on both training and fine-tuning an NMT model. Fine-tuning is a common practice in optimizing an MT model with respect to new data. However, new data potentially introduces new words (or tokens), which, if not taken into consideration, may lead to suboptimal performance. In addition, the distribution of tokens in the new data can differ from the distribution of the original data. As such, the original SW tokenization model could be less suitable for the new data. Through a systematic empirical evaluation, in this work we compare different strategies for SW tokenization and vocabulary generation with the ultimate goal to uncover an optimal setting for fine-tuning a domain-specific model. Furthermore, we developed several (in-domain) models, the best of which achieves 6 BLEU points improvement over the baseline.

Related papers

• A Bayesian Approach to Data Point Selection [24.98069363998565]
  Data point selection (DPS) is becoming a critical topic in deep learning. Existing approaches to DPS are predominantly based on a bi-level optimisation (BLO) formulation. We propose a novel Bayesian approach to DPS.
  arXiv  Detail & Related papers  (2024-11-06T09:04:13Z)
• Functional Graphical Models: Structure Enables Offline Data-Driven Optimization [111.28605744661638]
  We show how structure can enable sample-efficient data-driven optimization. We also present a data-driven optimization algorithm that infers the FGM structure itself.
  arXiv  Detail & Related papers  (2024-01-08T22:33:14Z)
• Unified Model Learning for Various Neural Machine Translation [63.320005222549646]
  Existing machine translation (NMT) studies mainly focus on developing dataset-specific models. We propose a versatile'' model, i.e., the Unified Model Learning for NMT (UMLNMT) that works with data from different tasks. OurNMT results in substantial improvements over dataset-specific models with significantly reduced model deployment costs.
  arXiv  Detail & Related papers  (2023-05-04T12:21:52Z)
• DaLC: Domain Adaptation Learning Curve Prediction for Neural Machine Translation [10.03007605098947]
  Domain Adaptation (DA) of Neural Machine Translation (NMT) model often relies on a pre-trained general NMT model which is adapted to the new domain on a sample of in-domain parallel data. We propose a Domain Learning Curve prediction (DaLC) model that predicts prospective DA performance based on in-domain monolingual samples in the source language.
  arXiv  Detail & Related papers  (2022-04-20T06:57:48Z)
• Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation [50.54059385277964]
  We present a novel data augmentation paradigm termed Continuous Semantic Augmentation (CsaNMT) CsaNMT augments each training instance with an adjacency region that could cover adequate variants of literal expression under the same meaning.
  arXiv  Detail & Related papers  (2022-04-14T08:16:28Z)
• Better Language Model with Hypernym Class Prediction [101.8517004687825]
  Class-based language models (LMs) have been long devised to address context sparsity in $n$-gram LMs. In this study, we revisit this approach in the context of neural LMs.
  arXiv  Detail & Related papers  (2022-03-21T01:16:44Z)
• Exploring Unsupervised Pretraining Objectives for Machine Translation [99.5441395624651]
  Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT) Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. We compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.
  arXiv  Detail & Related papers  (2021-06-10T10:18:23Z)
• Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey [9.645196221785694]
  We focus on robust approaches to domain adaptation for Neural Machine Translation (NMT) models. In particular, we look at the case where a system may need to translate sentences from multiple domains. We highlight the benefits of domain adaptation and multi-domain adaptation techniques to other lines of NMT research.
  arXiv  Detail & Related papers  (2021-04-14T16:21:37Z)
• Few-shot learning through contextual data augmentation [74.20290390065475]
  Machine translation models need to adapt to new data to maintain their performance over time. We show that adaptation on the scale of one to five examples is possible. Our model reports better accuracy scores than a reference system trained with on average 313 parallel examples.
  arXiv  Detail & Related papers  (2021-03-31T09:05:43Z)
• Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models [20.976165305749777]
  We learn a curriculum for improving a pre-trained NMT model by re-selecting influential data samples from the original training set. We propose a data selection framework based on Deterministic Actor-Critic, in which a critic network predicts the expected change of model performance.
  arXiv  Detail & Related papers  (2020-04-13T03:40:44Z)
• Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining [37.2106265998237]
  We propose an effective learning procedure named Meta Fine-Tuning (MFT) MFT serves as a meta-learner to solve a group of similar NLP tasks for neural language models. We implement MFT upon BERT to solve several multi-domain text mining tasks.
  arXiv  Detail & Related papers  (2020-03-29T11:27:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.