Transformers in Reinforcement Learning: A Survey

• URL: http://arxiv.org/abs/2307.05979v1
• Date: Wed, 12 Jul 2023 07:51:12 GMT
• Title: Transformers in Reinforcement Learning: A Survey
• Authors: Pranav Agarwal, Aamer Abdul Rahman, Pierre-Luc St-Charles, Simon J.D. Prince, Samira Ebrahimi Kahou
• Abstract summary: Transformers have impacted domains like natural language processing, computer vision, and robotics, where they improve performance compared to other neural networks. This survey explores how transformers are used in reinforcement learning (RL), where they are seen as a promising solution for addressing challenges such as unstable training, credit assignment, lack of interpretability, and partial observability.
• Score: 7.622978576824539
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Transformers have significantly impacted domains like natural language processing, computer vision, and robotics, where they improve performance compared to other neural networks. This survey explores how transformers are used in reinforcement learning (RL), where they are seen as a promising solution for addressing challenges such as unstable training, credit assignment, lack of interpretability, and partial observability. We begin by providing a brief domain overview of RL, followed by a discussion on the challenges of classical RL algorithms. Next, we delve into the properties of the transformer and its variants and discuss the characteristics that make them well-suited to address the challenges inherent in RL. We examine the application of transformers to various aspects of RL, including representation learning, transition and reward function modeling, and policy optimization. We also discuss recent research that aims to enhance the interpretability and efficiency of transformers in RL, using visualization techniques and efficient training strategies. Often, the transformer architecture must be tailored to the specific needs of a given application. We present a broad overview of how transformers have been adapted for several applications, including robotics, medicine, language modeling, cloud computing, and combinatorial optimization. We conclude by discussing the limitations of using transformers in RL and assess their potential for catalyzing future breakthroughs in this field.

Related papers

• A Survey of Techniques for Optimizing Transformer Inference [3.6258657276072253]
  Recent years have seen a phenomenal rise in performance and applications of transformer neural networks. Transformer-based networks such as ChatGPT have impacted the lives of common men. Researchers have proposed techniques to optimize transformer inference at all levels of abstraction.
  arXiv  Detail & Related papers  (2023-07-16T08:50:50Z)
• A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks [60.38369406877899]
  Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. Our survey encompasses the identification of the top five application domains for transformer-based models.
  arXiv  Detail & Related papers  (2023-06-11T23:13:51Z)
• Emergent Agentic Transformer from Chain of Hindsight Experience [96.56164427726203]
  We show that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches. This is the first time that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches.
  arXiv  Detail & Related papers  (2023-05-26T00:43:02Z)
• A Survey on Transformers in Reinforcement Learning [66.23773284875843]
  Transformer has been considered the dominating neural architecture in NLP and CV, mostly under supervised settings. Recently, a similar surge of using Transformers has appeared in the domain of reinforcement learning (RL), but it is faced with unique design choices and challenges brought by the nature of RL. This paper systematically reviews motivations and progress on using Transformers in RL, provide a taxonomy on existing works, discuss each sub-field, and summarize future prospects.
  arXiv  Detail & Related papers  (2023-01-08T14:04:26Z)
• On Transforming Reinforcement Learning by Transformer: The Development Trajectory [97.79247023389445]
  Transformer, originally devised for natural language processing, has also attested significant success in computer vision. We group existing developments in two categories: architecture enhancement and trajectory optimization. We examine the main applications of TRL in robotic manipulation, text-based games, navigation and autonomous driving.
  arXiv  Detail & Related papers  (2022-12-29T03:15:59Z)
• Vision Transformers for Action Recognition: A Survey [41.69370782177517]
  Vision transformers are emerging as a powerful tool to solve computer vision problems. Recent techniques have proven the efficacy of transformers beyond the image domain to solve numerous video-related tasks. Human action recognition is receiving special attention from the research community due to its widespread applications.
  arXiv  Detail & Related papers  (2022-09-13T02:57:05Z)
• Transformers in Vision: A Survey [101.07348618962111]
  Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence. Transformers require minimal inductive biases for their design and are naturally suited as set-functions. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline.
  arXiv  Detail & Related papers  (2021-01-04T18:57:24Z)
• A Survey on Visual Transformer [126.56860258176324]
  Transformer is a type of deep neural network mainly based on the self-attention mechanism. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages.
  arXiv  Detail & Related papers  (2020-12-23T09:37:54Z)
• Stabilizing Transformer-Based Action Sequence Generation For Q-Learning [5.707122938235432]
  The goal is a simple Transformer-based Deep Q-Learning method that is stable over several environments. The proposed method can match the performance of classic Q-learning on control environments while showing potential on some selected Atari benchmarks.
  arXiv  Detail & Related papers  (2020-10-23T22:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.