Hyper-Decision Transformer for Efficient Online Policy Adaptation

• URL: http://arxiv.org/abs/2304.08487v1
• Date: Mon, 17 Apr 2023 17:59:32 GMT
• Title: Hyper-Decision Transformer for Efficient Online Policy Adaptation
• Authors: Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, Chuang Gan
• Abstract summary: We propose a new framework, called Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a handful of demonstrations. We find that with a single expert demonstration and fine-tuning only 0.5% of DT parameters, HDT adapts faster to unseen tasks than fine-tuning the whole DT model.
• Score: 66.91294935068957
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Decision Transformers (DT) have demonstrated strong performances in offline reinforcement learning settings, but quickly adapting to unseen novel tasks remains challenging. To address this challenge, we propose a new framework, called Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a handful of demonstrations in a data- and parameter-efficient manner. To achieve such a goal, we propose to augment the base DT with an adaptation module, whose parameters are initialized by a hyper-network. When encountering unseen tasks, the hyper-network takes a handful of demonstrations as inputs and initializes the adaptation module accordingly. This initialization enables HDT to efficiently adapt to novel tasks by only fine-tuning the adaptation module. We validate HDT's generalization capability on object manipulation tasks. We find that with a single expert demonstration and fine-tuning only 0.5% of DT parameters, HDT adapts faster to unseen tasks than fine-tuning the whole DT model. Finally, we explore a more challenging setting where expert actions are not available, and we show that HDT outperforms state-of-the-art baselines in terms of task success rates by a large margin.

Related papers

• DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations [21.06471370479668]
  We introduce DiTASK, a novel approach to efficiently adapt pre-trained Vision Transformers for multiple tasks. Our theoretical analysis shows that DiTASK achieves full-rank updates during optimization, preserving the geometric structure of pre-trained features. Our experiments on PASCAL MTL and NYUD show that DiTASK achieves state-of-the-art performance across four dense prediction tasks, using 75% fewer parameters than existing methods.
  arXiv  Detail & Related papers  (2025-02-09T21:05:11Z)
• TADFormer : Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learning [14.888918165109244]
  Task-Efficient Dynamic transFormer, TADFormer, is a novel PEFT framework that performs task-aware feature adaptation in the fine-grained manner. TADFormer achieves higher accuracy in dense scene understanding tasks, while reducing the number of trainable parameters by up to 8.4 times.
  arXiv  Detail & Related papers  (2025-01-08T05:35:07Z)
• Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation [67.13876021157887]
  Dynamic Tuning (DyT) is a novel approach to improve both parameter and inference efficiency for ViT adaptation. DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.
  arXiv  Detail & Related papers  (2024-03-18T14:05:52Z)
• Task-Distributionally Robust Data-Free Meta-Learning [99.56612787882334]
  Data-Free Meta-Learning (DFML) aims to efficiently learn new tasks by leveraging multiple pre-trained models without requiring their original training data. For the first time, we reveal two major challenges hindering their practical deployments: Task-Distribution Shift ( TDS) and Task-Distribution Corruption (TDC)
  arXiv  Detail & Related papers  (2023-11-23T15:46:54Z)
• TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models [32.83440439290383]
  We introduce TAIL (Task-specific Adapters for Learning), a framework for efficient adaptation to new control tasks. Inspired by recent advancements in parameter-efficient fine-tuning in language domains, we explore efficient fine-tuning techniques. Our experiments in large-scale language-conditioned manipulation tasks suggest that TAIL with LoRA can achieve the best post-adaptation performance.
  arXiv  Detail & Related papers  (2023-10-09T17:49:50Z)
• Improved Test-Time Adaptation for Domain Generalization [48.239665441875374]
  Test-time training (TTT) adapts the learned model with test data. This work addresses two main factors: selecting an appropriate auxiliary TTT task for updating and identifying reliable parameters to update during the test phase. We introduce additional adaptive parameters for the trained model, and we suggest only updating the adaptive parameters during the test phase.
  arXiv  Detail & Related papers  (2023-04-10T10:12:38Z)
• Exploring Efficient Few-shot Adaptation for Vision Transformers [70.91692521825405]
  We propose a novel efficient Transformer Tuning (eTT) method that facilitates finetuning ViTs in the Few-shot Learning tasks. Key novelties come from the newly presented Attentive Prefix Tuning (APT) and Domain Residual Adapter (DRA) We conduct extensive experiments to show the efficacy of our model.
  arXiv  Detail & Related papers  (2023-01-06T08:42:05Z)
• Prompting Decision Transformer for Few-Shot Policy Generalization [98.0914217850999]
  We propose a Prompt-based Decision Transformer (Prompt-DT) to achieve few-shot adaptation in offline RL. Prompt-DT is a strong few-shot learner without any extra finetuning on unseen target tasks.
  arXiv  Detail & Related papers  (2022-06-27T17:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.