AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learning

• URL: http://arxiv.org/abs/2310.15719v2
• Date: Tue, 15 Oct 2024 17:14:26 GMT
• Title: AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learning
• Authors: Subhojeet Pramanik, Esraa Elelimy, Marlos C. Machado, Adam White,
• Abstract summary: We introduce alternatives to the transformer self-attention mechanism that offer context-independent inference cost. Compared with a state-of-the-art architecture, GTrXL, inference in our approach is at least 40% cheaper while reducing memory use more than 50%.
• Score: 7.886461196772644
• License:
• Abstract: In this paper we investigate transformer architectures designed for partially observable online reinforcement learning. The self-attention mechanism in the transformer architecture is capable of capturing long-range dependencies and it is the main reason behind its effectiveness in processing sequential data. Nevertheless, despite their success, transformers have two significant drawbacks that still limit their applicability in online reinforcement learning: (1) in order to remember all past information, the self-attention mechanism requires access to the whole history to be provided as context. (2) The inference cost in transformers is expensive. In this paper, we introduce recurrent alternatives to the transformer self-attention mechanism that offer context-independent inference cost, leverage long-range dependencies effectively, and performs well in online reinforcement learning task. We quantify the impact of the different components of our architecture in a diagnostic environment and assess performance gains in 2D and 3D pixel-based partially-observable environments (e.g. T-Maze, Mystery Path, Craftax, and Memory Maze). Compared with a state-of-the-art architecture, GTrXL, inference in our approach is at least 40% cheaper while reducing memory use more than 50%. Our approach either performs similarly or better than GTrXL, improving more than 37% upon GTrXL performance in harder tasks.

Related papers

• Skip-Layer Attention: Bridging Abstract and Detailed Dependencies in Transformers [56.264673865476986]
  This paper introduces Skip-Layer Attention (SLA) to enhance Transformer models. SLA improves the model's ability to capture dependencies between high-level abstract features and low-level details. Our implementation extends the Transformer's functionality by enabling queries in a given layer to interact with keys and values from both the current layer and one preceding layer.
  arXiv  Detail & Related papers  (2024-06-17T07:24:38Z)
• Symmetric Dot-Product Attention for Efficient Training of BERT Language Models [5.838117137253223]
  We propose an alternative compatibility function for the self-attention mechanism introduced by the Transformer architecture. When applied to the pre-training of BERT-like models, this new symmetric attention mechanism reaches a score of 79.36 on the GLUE benchmark against 78.74 for the traditional implementation.
  arXiv  Detail & Related papers  (2024-06-10T15:24:15Z)
• Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers [1.1499643186017316]
  We propose Cross-Architecture Transfer Learning (XATL) to improve efficiency of Transformer Language Models. Methodabbr significantly reduces the training time up to 2.5x times and converges to a better minimum with up to 2.6% stronger model on the LM benchmarks within the same compute budget.
  arXiv  Detail & Related papers  (2024-04-03T12:27:36Z)
• Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems [27.781785405875084]
  We propose to leverage a Transformer-based architecture with attention layers to automatically capture feature interactions. We identify two key challenges for applying the vanilla Transformer architecture to web-scale recommender systems.
  arXiv  Detail & Related papers  (2023-11-10T05:57:57Z)
• Recurrent Action Transformer with Memory [39.58317527488534]
  This paper proposes a novel model architecture that incorporates a recurrent memory mechanism designed to regulate information retention. We conduct experiments on memory-intensive environments (ViZDoom-Two-Colors, T-Maze, Memory Maze, Minigrid-Memory), classic Atari games, and MuJoCo control environments. The results show that using memory can significantly improve performance in memory-intensive environments, while maintaining or improving results in classic environments.
  arXiv  Detail & Related papers  (2023-06-15T19:29:08Z)
• Dual Vision Transformer [114.1062057736447]
  We propose a novel Transformer architecture that aims to mitigate the cost issue, named Dual Vision Transformer (Dual-ViT) The new architecture incorporates a critical semantic pathway that can more efficiently compress token vectors into global semantics with reduced order of complexity. We empirically demonstrate that Dual-ViT provides superior accuracy than SOTA Transformer architectures with reduced training complexity.
  arXiv  Detail & Related papers  (2022-07-11T16:03:44Z)
• Transformers are Meta-Reinforcement Learners [0.060917028769172814]
  We present TrMRL, a meta-RL agent that mimics the memory reinstatement mechanism using the transformer architecture. We show that the self-attention computes a consensus representation that minimizes the Bayes Risk at each layer. Results show that TrMRL presents comparable or superior performance, sample efficiency, and out-of-distribution generalization.
  arXiv  Detail & Related papers  (2022-06-14T06:21:13Z)
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition [78.07924262215181]
  We introduce AdaViT, an adaptive framework that learns to derive usage policies on which patches, self-attention heads and transformer blocks to use. Our method obtains more than 2x improvement on efficiency compared to state-of-the-art vision transformers with only 0.8% drop of accuracy.
  arXiv  Detail & Related papers  (2021-11-30T18:57:02Z)
• Efficient pre-training objectives for Transformers [84.64393460397471]
  We study several efficient pre-training objectives for Transformers-based models. We prove that eliminating the MASK token and considering the whole output during the loss are essential choices to improve performance.
  arXiv  Detail & Related papers  (2021-04-20T00:09:37Z)
• Transformers Solve the Limited Receptive Field for Monocular Depth Prediction [82.90445525977904]
  We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers. This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
  arXiv  Detail & Related papers  (2021-03-22T18:00:13Z)
• Transformers with Competitive Ensembles of Independent Mechanisms [97.93090139318294]
  We propose a new Transformer layer which divides the hidden representation and parameters into multiple mechanisms, which only exchange information through attention. We study TIM on a large-scale BERT model, on the Image Transformer, and on speech enhancement and find evidence for semantically meaningful specialization as well as improved performance.
  arXiv  Detail & Related papers  (2021-02-27T21:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.