Enhancing Pre-Trained Decision Transformers with Prompt-Tuning Bandits

• URL: http://arxiv.org/abs/2502.04979v2
• Date: Mon, 10 Feb 2025 10:48:31 GMT
• Title: Enhancing Pre-Trained Decision Transformers with Prompt-Tuning Bandits
• Authors: Finn Rietz, Oleg Smirnov, Sara Karimi, Lele Cao,
• Abstract summary: We introduce a scalable bandit-based prompt-tuning method that learns to construct high-performance trajectory prompts. Our approach significantly enhances downstream task performance without modifying the pre-trained Transformer backbone.
• Score: 2.6731152954002924
• License:
• Abstract: Harnessing large offline datasets is vital for training foundation models that can generalize across diverse tasks. Offline Reinforcement Learning (RL) offers a powerful framework for these scenarios, enabling the derivation of optimal policies even from suboptimal data. The Prompting Decision Transformer (PDT) is an offline RL multi-task model that distinguishes tasks through stochastic trajectory prompts, which are task-specific tokens maintained in context during rollouts. However, PDT samples these tokens uniformly at random from per-task demonstration datasets, failing to account for differences in token informativeness and potentially leading to performance degradation. To address this limitation, we introduce a scalable bandit-based prompt-tuning method that dynamically learns to construct high-performance trajectory prompts. Our approach significantly enhances downstream task performance without modifying the pre-trained Transformer backbone. Empirical results on benchmark tasks and a newly designed multi-task environment demonstrate the effectiveness of our method, creating a seamless bridge between general multi-task offline pre-training and task-specific online adaptation.

Related papers

• Towards bandit-based prompt-tuning for in-the-wild foundation agents [2.6731152954002924]
  We propose an inference time bandit-based prompt-tuning framework to enhance task performance. Our experiments indicate not only clear performance gains due to bandit-based prompt-tuning, but also better sample complexity, scalability, and prompt space exploration.
  arXiv  Detail & Related papers  (2025-02-10T11:20:10Z)
• Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer [10.338170161831496]
  Decision Transformer (DT) has emerged as a promising class of algorithms in offline reinforcement learning (RL) tasks. We introduce the Language model-d Prompt Transformer (LPDT), which leverages pre-trained language models for meta-RL tasks and fine-tunes the model using Low-rank Adaptation (LoRA) Our approach integrates pre-trained language model and RL tasks seamlessly.
  arXiv  Detail & Related papers  (2024-08-02T17:25:34Z)
• Offline Reinforcement Learning from Datasets with Structured Non-Stationarity [50.35634234137108]
  Current Reinforcement Learning (RL) is often limited by the large amount of data needed to learn a successful policy. We address a novel Offline RL problem setting in which, while collecting the dataset, the transition and reward functions gradually change between episodes but stay constant within each episode. We propose a method based on Contrastive Predictive Coding that identifies this non-stationarity in the offline dataset, accounts for it when training a policy, and predicts it during evaluation.
  arXiv  Detail & Related papers  (2024-05-23T02:41:36Z)
• Task-Distributionally Robust Data-Free Meta-Learning [99.56612787882334]
  Data-Free Meta-Learning (DFML) aims to efficiently learn new tasks by leveraging multiple pre-trained models without requiring their original training data. For the first time, we reveal two major challenges hindering their practical deployments: Task-Distribution Shift ( TDS) and Task-Distribution Corruption (TDC)
  arXiv  Detail & Related papers  (2023-11-23T15:46:54Z)
• Supervised Pretraining Can Learn In-Context Reinforcement Learning [96.62869749926415]
  In this paper, we study the in-context learning capabilities of transformers in decision-making problems. We introduce and study Decision-Pretrained Transformer (DPT), a supervised pretraining method where the transformer predicts an optimal action. We find that the pretrained transformer can be used to solve a range of RL problems in-context, exhibiting both exploration online and conservatism offline.
  arXiv  Detail & Related papers  (2023-06-26T17:58:50Z)
• Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
  We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
  arXiv  Detail & Related papers  (2023-06-06T02:24:41Z)
• Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving [103.745551954983]
  In this paper, we investigate the transfer performance of various types of self-supervised methods, including MoCo and SimCLR, on three downstream tasks. We find that their performances are sub-optimal or even lag far behind the single-task baseline. We propose a simple yet effective pretrain-adapt-finetune paradigm for general multi-task training.
  arXiv  Detail & Related papers  (2022-09-19T12:15:31Z)
• Prompting Decision Transformer for Few-Shot Policy Generalization [98.0914217850999]
  We propose a Prompt-based Decision Transformer (Prompt-DT) to achieve few-shot adaptation in offline RL. Prompt-DT is a strong few-shot learner without any extra finetuning on unseen target tasks.
  arXiv  Detail & Related papers  (2022-06-27T17:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.