Decoder-only Streaming Transformer for Simultaneous Translation

• URL: http://arxiv.org/abs/2406.03878v1
• Date: Thu, 6 Jun 2024 09:13:13 GMT
• Title: Decoder-only Streaming Transformer for Simultaneous Translation
• Authors: Shoutao Guo, Shaolei Zhang, Yang Feng,
• Abstract summary: Simultaneous Machine Translation (SiMT) generates translation while reading source tokens, essentially producing the target prefix based on the source prefix. We explore the potential of Decoder-only architecture, owing to its superior performance in various tasks and its inherent compatibility with SiMT. We propose the first Decoder-only SiMT model, named Decoder-only Streaming Transformer (DST)
• Score: 31.558179590071973
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Simultaneous Machine Translation (SiMT) generates translation while reading source tokens, essentially producing the target prefix based on the source prefix. To achieve good performance, it leverages the relationship between source and target prefixes to exact a policy to guide the generation of translations. Although existing SiMT methods primarily focus on the Encoder-Decoder architecture, we explore the potential of Decoder-only architecture, owing to its superior performance in various tasks and its inherent compatibility with SiMT. However, directly applying the Decoder-only architecture to SiMT poses challenges in terms of training and inference. To alleviate the above problems, we propose the first Decoder-only SiMT model, named Decoder-only Streaming Transformer (DST). Specifically, DST separately encodes the positions of the source and target prefixes, ensuring that the position of the target prefix remains unaffected by the expansion of the source prefix. Furthermore, we propose a Streaming Self-Attention (SSA) mechanism tailored for the Decoder-only architecture. It is capable of obtaining translation policy by assessing the sufficiency of input source information and integrating with the soft-attention mechanism to generate translations. Experiments demonstrate that our approach achieves state-of-the-art performance on three translation tasks.

Related papers

• Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder [75.03283861464365]
  The seq2seq task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task.
  arXiv  Detail & Related papers  (2023-04-08T15:44:29Z)
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders [92.90543340071007]
  We introduce DeltaLM, a pretrained multilingual encoder-decoder model. Specifically, we augment the pretrained multilingual encoder with a decoder and pre-train it in a self-supervised way. Experiments show that DeltaLM outperforms various strong baselines on both natural language generation and translation tasks.
  arXiv  Detail & Related papers  (2021-06-25T16:12:10Z)
• Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders [30.160261563657947]
  Speech-to-translation data is scarce; pre-training is promising in end-to-end Speech Translation. We propose a Stacked. Acoustic-and-Textual (SATE) method for speech translation. Our encoder begins with processing the acoustic sequence as usual, but later behaves more like an. MT encoder for a global representation of the input sequence.
  arXiv  Detail & Related papers  (2021-05-12T16:09:53Z)
• AlloST: Low-resource Speech Translation without Source Transcription [17.53382405899421]
  We propose a learning framework that utilizes a language-independent universal phone recognizer. The framework is based on an attention-based sequence-to-sequence model. Experiments conducted on the Fisher Spanish-English and Taigi-Mandarin drama corpora show that our method outperforms the conformer-based baseline.
  arXiv  Detail & Related papers  (2021-05-01T05:30:18Z)
• On the Sub-Layer Functionalities of Transformer Decoder [74.83087937309266]
  We study how Transformer-based decoders leverage information from the source and target languages. Based on these insights, we demonstrate that the residual feed-forward module in each Transformer decoder layer can be dropped with minimal loss of performance.
  arXiv  Detail & Related papers  (2020-10-06T11:50:54Z)
• Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation [127.54315184545796]
  Speech translation (ST) aims to learn transformations from speech in the source language to the text in the target language. We propose to improve the multitask ST model by utilizing word embedding as the intermediate.
  arXiv  Detail & Related papers  (2020-05-21T14:22:35Z)
• Bi-Decoder Augmented Network for Neural Machine Translation [108.3931242633331]
  We propose a novel Bi-Decoder Augmented Network (BiDAN) for the neural machine translation task. Since each decoder transforms the representations of the input text into its corresponding language, jointly training with two target ends can make the shared encoder has the potential to produce a language-independent semantic space.
  arXiv  Detail & Related papers  (2020-01-14T02:05:14Z)
• Explicit Sentence Compression for Neural Machine Translation [110.98786673598016]
  State-of-the-art Transformer-based neural machine translation (NMT) systems still follow a standard encoder-decoder framework. backbone information, which stands for the gist of a sentence, is not specifically focused on. We propose an explicit sentence compression method to enhance the source sentence representation for NMT.
  arXiv  Detail & Related papers  (2019-12-27T04:14:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.