Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making
- URL: http://arxiv.org/abs/2310.03022v3
- Date: Thu, 30 May 2024 07:19:34 GMT
- Title: Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making
- Authors: Jeonghye Kim, Suyoung Lee, Woojun Kim, Youngchul Sung,
- Abstract summary: Decision Transformer (DT) is emerging as a promising model based on Transformer.
We propose a novel action sequence predictor, named Decision ConvFormer (DC), based on the architecture of MetaFormer.
DC achieves state-the-art performance across various standard RL benchmarks while requiring fewer resources.
- Score: 20.07425661382103
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The recent success of Transformer in natural language processing has sparked its use in various domains. In offline reinforcement learning (RL), Decision Transformer (DT) is emerging as a promising model based on Transformer. However, we discovered that the attention module of DT is not appropriate to capture the inherent local dependence pattern in trajectories of RL modeled as a Markov decision process. To overcome the limitations of DT, we propose a novel action sequence predictor, named Decision ConvFormer (DC), based on the architecture of MetaFormer, which is a general structure to process multiple entities in parallel and understand the interrelationship among the multiple entities. DC employs local convolution filtering as the token mixer and can effectively capture the inherent local associations of the RL dataset. In extensive experiments, DC achieved state-of-the-art performance across various standard RL benchmarks while requiring fewer resources. Furthermore, we show that DC better understands the underlying meaning in data and exhibits enhanced generalization capability.
Related papers
- Critic-Guided Decision Transformer for Offline Reinforcement Learning [28.211835303617118]
Critic-Guided Decision Transformer (CGDT)
Uses predictability of long-term returns from value-based methods with the trajectory modeling capability of the Decision Transformer.
Builds upon these insights, we propose a novel approach, which combines the predictability of long-term returns from value-based methods with the trajectory modeling capability of the Decision Transformer.
arXiv Detail & Related papers (2023-12-21T10:29:17Z) - Rethinking Decision Transformer via Hierarchical Reinforcement Learning [54.3596066989024]
Decision Transformer (DT) is an innovative algorithm leveraging recent advances of the transformer architecture in reinforcement learning (RL)
We introduce a general sequence modeling framework for studying sequential decision making through the lens of Hierarchical RL.
We show DT emerges as a special case of this framework with certain choices of high-level and low-level policies, and discuss the potential failure of these choices.
arXiv Detail & Related papers (2023-11-01T03:32:13Z) - Supervised Pretraining Can Learn In-Context Reinforcement Learning [96.62869749926415]
In this paper, we study the in-context learning capabilities of transformers in decision-making problems.
We introduce and study Decision-Pretrained Transformer (DPT), a supervised pretraining method where the transformer predicts an optimal action.
We find that the pretrained transformer can be used to solve a range of RL problems in-context, exhibiting both exploration online and conservatism offline.
arXiv Detail & Related papers (2023-06-26T17:58:50Z) - On Transforming Reinforcement Learning by Transformer: The Development
Trajectory [97.79247023389445]
Transformer, originally devised for natural language processing, has also attested significant success in computer vision.
We group existing developments in two categories: architecture enhancement and trajectory optimization.
We examine the main applications of TRL in robotic manipulation, text-based games, navigation and autonomous driving.
arXiv Detail & Related papers (2022-12-29T03:15:59Z) - Explaining Cross-Domain Recognition with Interpretable Deep Classifier [100.63114424262234]
Interpretable Deep (IDC) learns the nearest source samples of a target sample as evidence upon which the classifier makes the decision.
Our IDC leads to a more explainable model with almost no accuracy degradation and effectively calibrates classification for optimum reject options.
arXiv Detail & Related papers (2022-11-15T15:58:56Z) - MoCoDA: Model-based Counterfactual Data Augmentation [40.878444530293635]
We argue that the ability to recognize and use local factorization in transition dynamics is a key element in unlocking the power of multi-object reasoning.
Knowing the local structure also allows us to predict which unseen states and actions this dynamics model will generalize to.
We show that MoCoDA enables RL agents to learn policies that generalize to unseen states and actions.
arXiv Detail & Related papers (2022-10-20T14:09:48Z) - Towards Unsupervised Domain Adaptation via Domain-Transformer [0.0]
We propose the Domain-Transformer (DoT) for Unsupervised Domain Adaptation (UDA)
DoT integrates the CNN-backbones and the core attention mechanism of Transformers from a new perspective.
It achieves the local semantic consistency across domains, where the domain-level attention and manifold regularization are explored.
arXiv Detail & Related papers (2022-02-24T02:30:15Z) - Generalized Decision Transformer for Offline Hindsight Information
Matching [16.7594941269479]
We present Generalized Decision Transformer (GDT) for solving any hindsight information matching (HIM) problem.
We show how different choices for the feature function and the anti-causal aggregator lead to novel Categorical DT (CDT) and Bi-directional DT (BDT) for matching different statistics of the future.
arXiv Detail & Related papers (2021-11-19T18:56:13Z) - HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain
Language Model Compression [53.90578309960526]
Large pre-trained language models (PLMs) have shown overwhelming performances compared with traditional neural network methods.
We propose a hierarchical relational knowledge distillation (HRKD) method to capture both hierarchical and domain relational information.
arXiv Detail & Related papers (2021-10-16T11:23:02Z) - Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
State-of-the-art neural language models (LMs) represented by Transformers are highly complex.
This paper proposes a full Bayesian learning framework for Transformer LM estimation.
arXiv Detail & Related papers (2021-02-09T10:55:27Z) - Counterfactual Data Augmentation using Locally Factored Dynamics [44.37487079747397]
Local causal structures can be leveraged to improve the sample efficiency of sequence prediction and off-policy reinforcement learning.
We propose an approach to inferring these structures given an object-oriented state representation, as well as a novel algorithm for Counterfactual Data Augmentation.
arXiv Detail & Related papers (2020-07-06T16:29:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.