Decision Transformer as a Foundation Model for Partially Observable Continuous Control

• URL: http://arxiv.org/abs/2404.02407v1
• Date: Wed, 3 Apr 2024 02:17:34 GMT
• Title: Decision Transformer as a Foundation Model for Partially Observable Continuous Control
• Authors: Xiangyuan Zhang, Weichao Mao, Haoran Qiu, Tamer Başar,
• Abstract summary: Decision Transformer (DT) architecture is used to predict optimal action based on past observations, actions, and rewards. DT exhibits remarkable zero-shot generalization abilities for completely new tasks. These findings highlight the potential of DT as a foundational controller for general control applications.
• Score: 5.453548045211778
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Closed-loop control of nonlinear dynamical systems with partial-state observability demands expert knowledge of a diverse, less standardized set of theoretical tools. Moreover, it requires a delicate integration of controller and estimator designs to achieve the desired system behavior. To establish a general controller synthesis framework, we explore the Decision Transformer (DT) architecture. Specifically, we first frame the control task as predicting the current optimal action based on past observations, actions, and rewards, eliminating the need for a separate estimator design. Then, we leverage the pre-trained language models, i.e., the Generative Pre-trained Transformer (GPT) series, to initialize DT and subsequently train it for control tasks using low-rank adaptation (LoRA). Our comprehensive experiments across five distinct control tasks, ranging from maneuvering aerospace systems to controlling partial differential equations (PDEs), demonstrate DT's capability to capture the parameter-agnostic structures intrinsic to control tasks. DT exhibits remarkable zero-shot generalization abilities for completely new tasks and rapidly surpasses expert performance levels with a minimal amount of demonstration data. These findings highlight the potential of DT as a foundational controller for general control applications.

Related papers

• ReACT: Reinforcement Learning for Controller Parametrization using B-Spline Geometries [0.0]
  This work presents a novel approach using deep reinforcement learning (DRL) with N-dimensional B-spline geometries (BSGs) We focus on the control of parameter-variant systems, a class of systems with complex behavior which depends on the operating conditions. We make the adaptation process more efficient by introducing BSGs to map the controller parameters which may depend on numerous operating conditions.
  arXiv  Detail & Related papers  (2024-01-10T16:27:30Z)
• TLControl: Trajectory and Language Control for Human Motion Synthesis [68.09806223962323]
  We present TLControl, a novel method for realistic human motion synthesis. It incorporates both low-level Trajectory and high-level Language semantics controls. It is practical for interactive and high-quality animation generation.
  arXiv  Detail & Related papers  (2023-11-28T18:54:16Z)
• Fine-Tuning Language Models Using Formal Methods Feedback [53.24085794087253]
  We present a fully automated approach to fine-tune pre-trained language models for applications in autonomous systems. The method synthesizes automaton-based controllers from pre-trained models guided by natural language task descriptions. The results indicate an improvement in percentage of specifications satisfied by the controller from 60% to 90%.
  arXiv  Detail & Related papers  (2023-10-27T16:24:24Z)
• SMART: Self-supervised Multi-task pretrAining with contRol Transformers [34.604339091596884]
  Self-supervised pretraining has been extensively studied in language and vision domains. It is difficult to properly design such a pretraining approach for sequential decision-making tasks. We propose a generic pretraining framework for sequential decision making.
  arXiv  Detail & Related papers  (2023-01-24T05:01:23Z)
• Designing a Robust Low-Level Agnostic Controller for a Quadrotor with Actor-Critic Reinforcement Learning [0.38073142980732994]
  We introduce domain randomization during the training phase of a low-level waypoint guidance controller based on Soft Actor-Critic. We show that, by introducing a certain degree of uncertainty in quadrotor dynamics during training, we can obtain a controller that is capable to perform the proposed task using a larger variation of quadrotor parameters.
  arXiv  Detail & Related papers  (2022-10-06T14:58:19Z)
• Performance-Driven Controller Tuning via Derivative-Free Reinforcement Learning [6.5158195776494]
  We tackle the controller tuning problem using a novel derivative-free reinforcement learning framework. We conduct numerical experiments on two concrete examples from autonomous driving, namely, adaptive cruise control with PID controller and trajectory tracking with MPC controller. Experimental results show that the proposed method outperforms popular baselines and highlight its strong potential for controller tuning.
  arXiv  Detail & Related papers  (2022-09-11T13:01:14Z)
• Automatic Rule Induction for Efficient Semi-Supervised Learning [56.91428251227253]
  Semi-supervised learning has shown promise in allowing NLP models to generalize from small amounts of labeled data. Pretrained transformer models act as black-box correlation engines that are difficult to explain and sometimes behave unreliably. We propose tackling both of these challenges via Automatic Rule Induction (ARI), a simple and general-purpose framework.
  arXiv  Detail & Related papers  (2022-05-18T16:50:20Z)
• Steady-State Error Compensation in Reference Tracking and Disturbance Rejection Problems for Reinforcement Learning-Based Control [0.9023847175654602]
  Reinforcement learning (RL) is a promising, upcoming topic in automatic control applications. Initiative action state augmentation (IASA) for actor-critic-based RL controllers is introduced. This augmentation does not require any expert knowledge, leaving the approach model free.
  arXiv  Detail & Related papers  (2022-01-31T16:29:19Z)
• Sparsity in Partially Controllable Linear Systems [56.142264865866636]
  We study partially controllable linear dynamical systems specified by an underlying sparsity pattern. Our results characterize those state variables which are irrelevant for optimal control.
  arXiv  Detail & Related papers  (2021-10-12T16:41:47Z)
• Improving Input-Output Linearizing Controllers for Bipedal Robots via Reinforcement Learning [85.13138591433635]
  The main drawbacks of input-output linearizing controllers are the need for precise dynamics models and not being able to account for input constraints. In this paper, we address both challenges for the specific case of bipedal robot control by the use of reinforcement learning techniques.
  arXiv  Detail & Related papers  (2020-04-15T18:15:49Z)
• Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives [89.34229413345541]
  We propose a conditioning scheme which avoids pitfalls by learning the controller and its conditioning in an end-to-end manner. Our model predicts complex action sequences based directly on a dynamic image representation of the robot motion. We report significant improvements in task success over representative MPC and IL baselines.
  arXiv  Detail & Related papers  (2020-03-19T15:04:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.