GRET: Global Representation Enhanced Transformer

• URL: http://arxiv.org/abs/2002.10101v1
• Date: Mon, 24 Feb 2020 07:37:17 GMT
• Title: GRET: Global Representation Enhanced Transformer
• Authors: Rongxiang Weng, Haoran Wei, Shujian Huang, Heng Yu, Lidong Bing, Weihua Luo, Jiajun Chen
• Abstract summary: Transformer, based on the encoder-decoder framework, has achieved state-of-the-art performance on several natural language generation tasks. We propose a novel global representation enhanced Transformer (GRET) to explicitly model global representation in the Transformer network.
• Score: 85.58930151690336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer, based on the encoder-decoder framework, has achieved state-of-the-art performance on several natural language generation tasks. The encoder maps the words in the input sentence into a sequence of hidden states, which are then fed into the decoder to generate the output sentence. These hidden states usually correspond to the input words and focus on capturing local information. However, the global (sentence level) information is seldom explored, leaving room for the improvement of generation quality. In this paper, we propose a novel global representation enhanced Transformer (GRET) to explicitly model global representation in the Transformer network. Specifically, in the proposed model, an external state is generated for the global representation from the encoder. The global representation is then fused into the decoder during the decoding process to improve generation quality. We conduct experiments in two text generation tasks: machine translation and text summarization. Experimental results on four WMT machine translation tasks and LCSTS text summarization task demonstrate the effectiveness of the proposed approach on natural language generation.

Related papers

• Other Tokens Matter: Exploring Global and Local Features of Vision Transformers for Object Re-Identification [63.147482497821166]
  We first explore the influence of global and local features of ViT and then propose a novel Global-Local Transformer (GLTrans) for high-performance object Re-ID. Our proposed method achieves superior performance on four object Re-ID benchmarks.
  arXiv  Detail & Related papers  (2024-04-23T12:42:07Z)
• Calibration & Reconstruction: Deep Integrated Language for Referring Image Segmentation [8.225408779913712]
  Referring image segmentation aims to segment an object referred to by natural language expression from an image. Conventional transformer decoders can distort linguistic information with deeper layers, leading to suboptimal results. We introduce CRFormer, a model that iteratively calibrates multi-modal features in the transformer decoder.
  arXiv  Detail & Related papers  (2024-04-12T07:13:32Z)
• Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training [120.91411454661741]
  We present a pre-trainable Universal-DEcoder Network (Uni-EDEN) to facilitate both vision-language perception and generation. Uni-EDEN is a two-stream Transformer based structure, consisting of three modules: object and sentence encoders that separately learns the representations of each modality.
  arXiv  Detail & Related papers  (2022-01-11T16:15:07Z)
• Sentence Bottleneck Autoencoders from Transformer Language Models [53.350633961266375]
  We build a sentence-level autoencoder from a pretrained, frozen transformer language model. We adapt the masked language modeling objective as a generative, denoising one, while only training a sentence bottleneck and a single-layer modified transformer decoder. We demonstrate that the sentence representations discovered by our model achieve better quality than previous methods that extract representations from pretrained transformers on text similarity tasks, style transfer, and single-sentence classification tasks in the GLUE benchmark, while using fewer parameters than large pretrained models.
  arXiv  Detail & Related papers  (2021-08-31T19:39:55Z)
• Transition based Graph Decoder for Neural Machine Translation [41.7284715234202]
  We propose a general Transformer-based approach for tree and graph decoding based on generating a sequence of transitions. We show improved performance over the standard Transformer decoder, as well as over ablated versions of the model.
  arXiv  Detail & Related papers  (2021-01-29T15:20:45Z)
• Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network [96.4761273757796]
  We introduce a Global Enhanced Transformer (termed GET) to enable the extraction of a more comprehensive global representation. GET adaptively guides the decoder to generate high-quality captions.
  arXiv  Detail & Related papers  (2020-12-13T13:38:58Z)
• Bi-Decoder Augmented Network for Neural Machine Translation [108.3931242633331]
  We propose a novel Bi-Decoder Augmented Network (BiDAN) for the neural machine translation task. Since each decoder transforms the representations of the input text into its corresponding language, jointly training with two target ends can make the shared encoder has the potential to produce a language-independent semantic space.
  arXiv  Detail & Related papers  (2020-01-14T02:05:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.