A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning

• URL: http://arxiv.org/abs/2407.03076v1
• Date: Wed, 3 Jul 2024 12:50:49 GMT
• Title: A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning
• Authors: Ramakrishna Appicharla, Baban Gain, Santanu Pal, Asif Ekbal, Pushpak Bhattacharyya,
• Abstract summary: In document-level neural machine translation (DocNMT), multi-encoder approaches are common in encoding context and source sentences. Recent studies have shown that the context encoder generates noise and makes the model robust to the choice of context. This paper further investigates this observation by explicitly modelling context encoding through multi-task learning (MTL) to make the model sensitive to the choice of context.
• Score: 49.62044186504516
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: In document-level neural machine translation (DocNMT), multi-encoder approaches are common in encoding context and source sentences. Recent studies \cite{li-etal-2020-multi-encoder} have shown that the context encoder generates noise and makes the model robust to the choice of context. This paper further investigates this observation by explicitly modelling context encoding through multi-task learning (MTL) to make the model sensitive to the choice of context. We conduct experiments on cascade MTL architecture, which consists of one encoder and two decoders. Generation of the source from the context is considered an auxiliary task, and generation of the target from the source is the main task. We experimented with German--English language pairs on News, TED, and Europarl corpora. Evaluation results show that the proposed MTL approach performs better than concatenation-based and multi-encoder DocNMT models in low-resource settings and is sensitive to the choice of context. However, we observe that the MTL models are failing to generate the source from the context. These observations align with the previous studies, and this might suggest that the available document-level parallel corpora are not context-aware, and a robust sentence-level model can outperform the context-aware models.

Related papers

• A Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation [20.120962279327493]
  We evaluate the models on the ContraPro test set to study how different contexts affect pronoun translation accuracy. Our analysis shows that the context encoder provides sufficient information to learn discourse-level information.
  arXiv  Detail & Related papers  (2023-08-11T10:35:53Z)
• On Search Strategies for Document-Level Neural Machine Translation [51.359400776242786]
  Document-level neural machine translation (NMT) models produce a more consistent output across a document. In this work, we aim to answer the question how to best utilize a context-aware translation model in decoding.
  arXiv  Detail & Related papers  (2023-06-08T11:30:43Z)
• HanoiT: Enhancing Context-aware Translation via Selective Context [95.93730812799798]
  Context-aware neural machine translation aims to use the document-level context to improve translation quality. The irrelevant or trivial words may bring some noise and distract the model from learning the relationship between the current sentence and the auxiliary context. We propose a novel end-to-end encoder-decoder model with a layer-wise selection mechanism to sift and refine the long document context.
  arXiv  Detail & Related papers  (2023-01-17T12:07:13Z)
• SMDT: Selective Memory-Augmented Neural Document Translation [53.4627288890316]
  We propose a Selective Memory-augmented Neural Document Translation model to deal with documents containing large hypothesis space of context. We retrieve similar bilingual sentence pairs from the training corpus to augment global context. We extend the two-stream attention model with selective mechanism to capture local context and diverse global contexts.
  arXiv  Detail & Related papers  (2022-01-05T14:23:30Z)
• Contrastive Learning for Context-aware Neural Machine TranslationUsing Coreference Information [14.671424999873812]
  We propose CorefCL, a novel data augmentation and contrastive learning scheme based on coreference between the source and contextual sentences. By corrupting automatically detected coreference mentions in the contextual sentence, CorefCL can train the model to be sensitive to coreference inconsistency. In experiments, our method consistently improved BLEU of compared models on English-German and English-Korean tasks.
  arXiv  Detail & Related papers  (2021-09-13T05:18:47Z)
• Divide and Rule: Training Context-Aware Multi-Encoder Translation Models with Little Resources [20.057692375546356]
  Multi-encoder models aim to improve translation quality by encoding document-level contextual information alongside the current sentence. We show that training these parameters takes large amount of data, since the contextual training signal is sparse. We propose an efficient alternative, based on splitting sentence pairs, that allows to enrich the training signal of a set of parallel sentences.
  arXiv  Detail & Related papers  (2021-03-31T15:15:32Z)
• Context-aware Decoder for Neural Machine Translation using a Target-side Document-Level Language Model [12.543106304662059]
  We present a method to turn a sentence-level translation model into a context-aware model by incorporating a document-level language model into the decoder. Our decoder is built upon only a sentence-level parallel corpora and monolingual corpora. In a theoretical viewpoint, the core part of this work is the novel representation of contextual information using point-wise mutual information between context and the current sentence.
  arXiv  Detail & Related papers  (2020-10-24T08:06:18Z)
• Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation [59.191079800436114]
  Document-level machine translation incorporates inter-sentential dependencies into the translation of a source sentence. We propose a new framework to model cross-sentence dependencies by training neural machine translation (NMT) to predict both the target translation and surrounding sentences of a source sentence.
  arXiv  Detail & Related papers  (2020-03-30T03:38:01Z)
• Towards Making the Most of Context in Neural Machine Translation [112.9845226123306]
  We argue that previous research did not make a clear use of the global context. We propose a new document-level NMT framework that deliberately models the local context of each sentence.
  arXiv  Detail & Related papers  (2020-02-19T03:30:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.