Dual Learning: Theoretical Study and an Algorithmic Extension

• URL: http://arxiv.org/abs/2005.08238v1
• Date: Sun, 17 May 2020 12:14:35 GMT
• Title: Dual Learning: Theoretical Study and an Algorithmic Extension
• Abstract summary: We propose multi-step dual learning, in which we leverage feedback signals from additional domains to improve the qualities of mappings. We prove that multi-step dual learn-ing can boost the performance of standard dual learning under mild conditions.
• Score: 168.0485041314289
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dual learning has been successfully applied in many machine learning applications including machine translation, image-to-image transformation, etc. The high-level idea of dual learning is very intuitive: if we map an $x$ from one domain to another and then map it back, we should recover the original $x$. Although its effectiveness has been empirically verified, theoretical understanding of dual learning is still very limited. In this paper, we aim at understanding why and when dual learning works. Based on our theoretical analysis, we further extend dual learning by introducing more related mappings and propose multi-step dual learning, in which we leverage feedback signals from additional domains to improve the qualities of the mappings. We prove that multi-step dual learn-ing can boost the performance of standard dual learning under mild conditions. Experiments on WMT 14 English$\leftrightarrow$German and MultiUNEnglish$\leftrightarrow$French translations verify our theoretical findings on dual learning, and the results on the translations among English, French, and Spanish of MultiUN demonstrate the effectiveness of multi-step dual learning.

Related papers

• Learning to Trace Seiberg Dualities [39.146761527401424]
  We use machine learning methods to address the question for Seiberg dualities of supersymmetric quiver gauge theories.<n>On the one hand, this leads us to a practical tool for establishing the computational complexity of different dualities.<n>We find that for quivers with a modest number of quiver nodes, different network architectures tend to outperform deterministic algorithms.
  arXiv  Detail & Related papers  (2026-07-30T17:59:56Z)
• Simulating LLM-to-LLM Tutoring for Multilingual Math Feedback [11.889826908536941]
  We present the first large-scale simulation of multilingual tutor-student interactions using large language models (LLMs)<n>A stronger model plays the role of the tutor, generating feedback in the form of hints, while a weaker model simulates the student.<n>Our study examines how student input language, teacher feedback language, model choice, and language resource level jointly influence performance.
  arXiv  Detail & Related papers  (2025-06-05T11:53:04Z)
• FonMTL: Towards Multitask Learning for the Fon Language [1.9370453715137865]
  We present the first explorative approach to multitask learning, for model capabilities enhancement in Natural Language Processing for the Fon language. We leverage two language model heads as encoders to build shared representations for the inputs, and we use linear layers blocks for classification relative to each task. Our results on the NER and POS tasks for Fon, show competitive (or better) performances compared to several multilingual pretrained language models finetuned on single tasks.
  arXiv  Detail & Related papers  (2023-08-28T03:26:21Z)
• MICO: A Multi-alternative Contrastive Learning Framework for Commonsense Knowledge Representation [52.238466443561705]
  MICO is a multi-alternative contrastve learning framework on COmmonsense knowledge graphs. It generates the commonsense knowledge representation by contextual interaction between entity nodes. It can benefit the following two tasks by simply comparing the distance score between the representations.
  arXiv  Detail & Related papers  (2022-10-14T06:51:21Z)
• Large-scale Bilingual Language-Image Contrastive Learning [17.19890778916312]
  We collect 1.1 billion image-text pairs (708 million Korean and 476 million English) and train a bilingual multimodal model named KELIP. We introduce simple yet effective training schemes, including MAE pre-training and multi-crop augmentation. Experiments demonstrate that a model trained with such training schemes shows competitive performance in both languages.
  arXiv  Detail & Related papers  (2022-03-28T03:02:03Z)
• The Reality of Multi-Lingual Machine Translation [3.183845608678763]
  "The Reality of Multi-Lingual Machine Translation" discusses the benefits and perils of using more than two languages in machine translation systems. Author: Machine translation is for us a prime example of deep learning applications.
  arXiv  Detail & Related papers  (2022-02-25T16:44:06Z)
• X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models [103.75890012041366]
  Language models (LMs) have proven surprisingly successful at capturing factual knowledge. However, studies on LMs' factual representation ability have almost invariably been performed on English. We create a benchmark of cloze-style probes for 23 typologically diverse languages.
  arXiv  Detail & Related papers  (2020-10-13T05:29:56Z)
• Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation [32.99075556718486]
  Iterative Back-Translation and Dual Learning effectively incorporate monolingual training data in neural machine translation, they use different objectives and gradient approximation strategies, and have not been extensively compared. We introduce a novel dual reconstruction objective that provides a unified view of Iterative Back-Translation and Dual Learning.
  arXiv  Detail & Related papers  (2020-10-07T13:40:32Z)
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training [135.12061144759517]
  We present an information-theoretic framework that formulates cross-lingual language model pre-training. We propose a new pre-training task based on contrastive learning. By leveraging both monolingual and parallel corpora, we jointly train the pretext to improve the cross-lingual transferability of pre-trained models.
  arXiv  Detail & Related papers  (2020-07-15T16:58:01Z)
• Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation [61.88012735215636]
  Unsupervised neural machine translation (UNMT) has recently achieved remarkable results for several language pairs. UNMT can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time. In this paper, we empirically introduce a simple method to translate between thirteen languages using a single encoder and a single decoder.
  arXiv  Detail & Related papers  (2020-04-21T17:26:16Z)
• Provable Representation Learning for Imitation Learning via Bi-level Optimization [60.059520774789654]
  A common strategy in modern learning systems is to learn a representation that is useful for many tasks. We study this strategy in the imitation learning setting for Markov decision processes (MDPs) where multiple experts' trajectories are available. We instantiate this framework for the imitation learning settings of behavior cloning and observation-alone.
  arXiv  Detail & Related papers  (2020-02-24T21:03:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.