Addressing Zero-Resource Domains Using Document-Level Context in Neural Machine Translation

• URL: http://arxiv.org/abs/2004.14927v2
• Date: Mon, 19 Apr 2021 11:25:20 GMT
• Title: Addressing Zero-Resource Domains Using Document-Level Context in Neural Machine Translation
• Authors: Dario Stojanovski, Alexander Fraser
• Abstract summary: We show that when in-domain parallel data is not available, access to document-level context enables better capturing of domain generalities. We present two document-level Transformer models which are capable of using large context sizes.
• Score: 80.40677540516616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Achieving satisfying performance in machine translation on domains for which there is no training data is challenging. Traditional supervised domain adaptation is not suitable for addressing such zero-resource domains because it relies on in-domain parallel data. We show that when in-domain parallel data is not available, access to document-level context enables better capturing of domain generalities compared to only having access to a single sentence. Having access to more information provides a more reliable domain estimation. We present two document-level Transformer models which are capable of using large context sizes and we compare these models against strong Transformer baselines. We obtain improvements for the two zero resource domains we study. We additionally provide an analysis where we vary the amount of context and look at the case where in-domain data is available.

Related papers

• M2D2: A Massively Multi-domain Language Modeling Dataset [76.13062203588089]
  We present M2D2, a fine-grained, massively multi-domain corpus for studying domain adaptation (LMs) Using categories derived from Wikipedia and ArXiv, we organize the domains in each data source into 22 groups. We show the benefits of adapting the LM along a domain hierarchy; adapting to smaller amounts of fine-grained domain-specific data can lead to larger in-domain performance gains.
  arXiv  Detail & Related papers  (2022-10-13T21:34:52Z)
• Domain-Specific Text Generation for Machine Translation [7.803471587734353]
  We propose a novel approach to domain adaptation leveraging state-of-the-art pretrained language models (LMs) for domain-specific data augmentation. We employ mixed fine-tuning to train models that significantly improve translation of in-domain texts.
  arXiv  Detail & Related papers  (2022-08-11T16:22:16Z)
• Translation Transformers Rediscover Inherent Data Domains [0.0]
  We analyze the sentence representations learned by NMT Transformers and show that these explicitly include the information on text domains. We show that this internal information is enough to cluster sentences by their underlying domains without supervision. We show that NMT models produce clusters better aligned to the actual domains compared to pre-trained language models (LMs)
  arXiv  Detail & Related papers  (2021-09-16T10:58:13Z)
• Data Augmentation for Cross-Domain Named Entity Recognition [22.66649873447105]
  We study cross-domain data augmentation for the named entity recognition task. We propose a novel neural architecture to transform the data representation from a high-resource to a low-resource domain. We show that transforming the data to the low-resource domain representation achieves significant improvements over only using data from high-resource domains.
  arXiv  Detail & Related papers  (2021-09-04T00:50:55Z)
• DINO: A Conditional Energy-Based GAN for Domain Translation [67.9879720396872]
  Domain translation is the process of transforming data from one domain to another while preserving the common semantics. Some of the most popular domain translation systems are based on conditional generative adversarial networks. We propose a new framework, where two networks are simultaneously trained, in a supervised manner, to perform domain translation in opposite directions.
  arXiv  Detail & Related papers  (2021-02-18T11:52:45Z)
• Domain2Vec: Domain Embedding for Unsupervised Domain Adaptation [56.94873619509414]
  Conventional unsupervised domain adaptation studies the knowledge transfer between a limited number of domains. We propose a novel Domain2Vec model to provide vectorial representations of visual domains based on joint learning of feature disentanglement and Gram matrix. We demonstrate that our embedding is capable of predicting domain similarities that match our intuition about visual relations between different domains.
  arXiv  Detail & Related papers  (2020-07-17T22:05:09Z)
• Domain Adaptation for Semantic Parsing [68.81787666086554]
  We propose a novel semantic for domain adaptation, where we have much fewer annotated data in the target domain compared to the source domain. Our semantic benefits from a two-stage coarse-to-fine framework, thus can provide different and accurate treatments for the two stages. Experiments on a benchmark dataset show that our method consistently outperforms several popular domain adaptation strategies.
  arXiv  Detail & Related papers  (2020-06-23T14:47:41Z)
• Unsupervised Domain Clusters in Pretrained Language Models [61.832234606157286]
  We show that massive pre-trained language models implicitly learn sentence representations that cluster by domains without supervision. We propose domain data selection methods based on such models. We evaluate our data selection methods for neural machine translation across five diverse domains.
  arXiv  Detail & Related papers  (2020-04-05T06:22:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.