A Survey of Transformers

• URL: http://arxiv.org/abs/2106.04554v1
• Date: Tue, 8 Jun 2021 17:43:08 GMT
• Title: A Survey of Transformers
• Authors: Tianyang Lin, Yuxin Wang, Xiangyang Liu, Xipeng Qiu
• Abstract summary: We introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications.
• Score: 31.701774010750086
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

Related papers

• Survey: Transformer-based Models in Data Modality Conversion [0.8136541584281987]
  Modality Conversion involves the transformation of data from one form of representation to another, mimicking the way humans integrate and interpret sensory information. This paper provides a comprehensive review of transformer-based models applied to the primary modalities of text, vision, and speech, discussing their architectures, conversion methodologies, and applications.
  arXiv  Detail & Related papers  (2024-08-08T18:39:14Z)
• Introduction to Transformers: an NLP Perspective [59.0241868728732]
  We introduce basic concepts of Transformers and present key techniques that form the recent advances of these models. This includes a description of the standard Transformer architecture, a series of model refinements, and common applications.
  arXiv  Detail & Related papers  (2023-11-29T13:51:04Z)
• An Introduction to Transformers [23.915718146956355]
  transformer is a neural network component that can be used to learn useful sequences or sets of data-points. In this note we aim for a mathematically precise, intuitive, and clean description of the transformer architecture.
  arXiv  Detail & Related papers  (2023-04-20T14:54:19Z)
• Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review [6.953789750981636]
  We provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks.
  arXiv  Detail & Related papers  (2023-01-09T16:56:23Z)
• A Survey on Transformers in Reinforcement Learning [66.23773284875843]
  Transformer has been considered the dominating neural architecture in NLP and CV, mostly under supervised settings. Recently, a similar surge of using Transformers has appeared in the domain of reinforcement learning (RL), but it is faced with unique design choices and challenges brought by the nature of RL. This paper systematically reviews motivations and progress on using Transformers in RL, provide a taxonomy on existing works, discuss each sub-field, and summarize future prospects.
  arXiv  Detail & Related papers  (2023-01-08T14:04:26Z)
• Transformers in Time Series: A Survey [66.50847574634726]
  We systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification.
  arXiv  Detail & Related papers  (2022-02-15T01:43:27Z)
• A Survey on Visual Transformer [126.56860258176324]
  Transformer is a type of deep neural network mainly based on the self-attention mechanism. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages.
  arXiv  Detail & Related papers  (2020-12-23T09:37:54Z)
• Efficient Transformers: A Survey [98.23264445730645]
  Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. This paper characterizes a large and thoughtful selection of recent efficiency-flavored "X-former" models.
  arXiv  Detail & Related papers  (2020-09-14T20:38:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.