You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models

• URL: http://arxiv.org/abs/2509.14031v1
• Date: Wed, 17 Sep 2025 14:33:17 GMT
• Title: You Are What You Train: Effects of Data Composition on Training Context-aware Machine Translation Models
• Authors: Paweł Mąka, Yusuf Can Semerci, Jan Scholtes, Gerasimos Spanakis,
• Abstract summary: Sparsity of contextually rich examples in the standard training data has been hypothesized as the reason for the difficulty of context utilization.<n>We demonstrate a strong association between training data sparsity and model performance confirming sparsity as a key bottleneck.<n>We propose and empirically evaluate two training strategies designed to leverage the available data.
• Score: 10.29311882711665
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Achieving human-level translations requires leveraging context to ensure coherence and handle complex phenomena like pronoun disambiguation. Sparsity of contextually rich examples in the standard training data has been hypothesized as the reason for the difficulty of context utilization. In this work, we systematically validate this claim in both single- and multilingual settings by constructing training datasets with a controlled proportions of contextually relevant examples. We demonstrate a strong association between training data sparsity and model performance confirming sparsity as a key bottleneck. Importantly, we reveal that improvements in one contextual phenomenon do no generalize to others. While we observe some cross-lingual transfer, it is not significantly higher between languages within the same sub-family. Finally, we propose and empirically evaluate two training strategies designed to leverage the available data. These strategies improve context utilization, resulting in accuracy gains of up to 6 and 8 percentage points on the ctxPro evaluation in single- and multilingual settings respectively.

Related papers

• Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning [23.900888224619]
  We show that curriculum learning consistently improves convergence in early and mid-training phases.<n>We identify compression ratio, lexical diversity, and readability as effective difficulty signals across settings.
  arXiv  Detail & Related papers  (2025-06-12T21:06:57Z)
• A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives [13.581385765600265]
  Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community. This paper proposes a comparison of multilingual pretraining objectives in a controlled methodological environment.
  arXiv  Detail & Related papers  (2024-07-22T09:16:30Z)
• Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data [13.587157318352869]
  We propose a two-phase training approach where pre-trained large language models are continually pre-trained on parallel data. We evaluate these methods on thirteen test sets for Japanese-to-English and English-to-Japanese translation.
  arXiv  Detail & Related papers  (2024-07-03T14:23:36Z)
• The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights [108.40766216456413]
  We propose a question alignment framework to bridge the gap between large language models' English and non-English performance. Experiment results show it can boost multilingual performance across diverse reasoning scenarios, model families, and sizes. We analyze representation space, generated response and data scales, and reveal how question translation training strengthens language alignment within LLMs.
  arXiv  Detail & Related papers  (2024-05-02T14:49:50Z)
• Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing [68.47787275021567]
  Cross-lingual semantic parsing transfers parsing capability from a high-resource language (e.g., English) to low-resource languages with scarce training data. We propose a new approach to cross-lingual semantic parsing by explicitly minimizing cross-lingual divergence between latent variables using Optimal Transport.
  arXiv  Detail & Related papers  (2023-07-09T04:52:31Z)
• Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation [49.916963624249355]
  A UNMT model is trained on the pseudo parallel data with translated source, and natural source sentences in inference. The source discrepancy between training and inference hinders the translation performance of UNMT models. We propose an online self-training approach, which simultaneously uses the pseudo parallel data natural source, translated target to mimic the inference scenario.
  arXiv  Detail & Related papers  (2022-03-16T04:50:27Z)
• Diverse Pretrained Context Encodings Improve Document Translation [31.03899564183553]
  We propose a new architecture for adapting a sentence-level sequence-to-sequence transformer incorporating multiple pretrained document context signals. Our best multi-context model consistently outperforms the best existing context-aware transformers.
  arXiv  Detail & Related papers  (2021-06-07T15:28:01Z)
• On the Language Coverage Bias for Neural Machine Translation [81.81456880770762]
  Language coverage bias is important for neural machine translation (NMT) because the target-original training data is not well exploited in current practice. By carefully designing experiments, we provide comprehensive analyses of the language coverage bias in the training data. We propose two simple and effective approaches to alleviate the language coverage bias problem.
  arXiv  Detail & Related papers  (2021-06-07T01:55:34Z)
• Few-shot learning through contextual data augmentation [74.20290390065475]
  Machine translation models need to adapt to new data to maintain their performance over time. We show that adaptation on the scale of one to five examples is possible. Our model reports better accuracy scores than a reference system trained with on average 313 parallel examples.
  arXiv  Detail & Related papers  (2021-03-31T09:05:43Z)
• Cross-lingual Spoken Language Understanding with Regularized Representation Alignment [71.53159402053392]
  We propose a regularization approach to align word-level and sentence-level representations across languages without any external resource. Experiments on the cross-lingual spoken language understanding task show that our model outperforms current state-of-the-art methods in both few-shot and zero-shot scenarios.
  arXiv  Detail & Related papers  (2020-09-30T08:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.