Elastic Decision Transformer

• URL: http://arxiv.org/abs/2307.02484v6
• Date: Fri, 20 Oct 2023 05:12:04 GMT
• Title: Elastic Decision Transformer
• Abstract summary: Elastic Decision Transformer (EDT) is a significant advancement over the existing Decision Transformer (DT) EDT facilitates trajectory stitching during action inference at test time, achieved by adjusting the history length maintained in DT. Extensive experimentation demonstrates EDT's ability to bridge the performance gap between DT-based and Q Learning-based approaches.
• Score: 18.085153645646646
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces Elastic Decision Transformer (EDT), a significant advancement over the existing Decision Transformer (DT) and its variants. Although DT purports to generate an optimal trajectory, empirical evidence suggests it struggles with trajectory stitching, a process involving the generation of an optimal or near-optimal trajectory from the best parts of a set of sub-optimal trajectories. The proposed EDT differentiates itself by facilitating trajectory stitching during action inference at test time, achieved by adjusting the history length maintained in DT. Further, the EDT optimizes the trajectory by retaining a longer history when the previous trajectory is optimal and a shorter one when it is sub-optimal, enabling it to "stitch" with a more optimal trajectory. Extensive experimentation demonstrates EDT's ability to bridge the performance gap between DT-based and Q Learning-based approaches. In particular, the EDT outperforms Q Learning-based methods in a multi-task regime on the D4RL locomotion benchmark and Atari games. Videos are available at: https://kristery.github.io/edt/

Related papers

• Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models [58.27409489926424]
  Fine-tuning instruct models often improve target performance while inducing behavioral drift from the reference model.<n>We show how behavioral drift induces a shared geometry anchored at the reference model, with the drift budget defining a boundary within this space.<n>Our results suggest that fine-tuning is not just about how much a model changes, but how that change is spent.
  arXiv  Detail & Related papers  (2026-09-12T03:13:54Z)
• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation [51.210887267509854]
  We argue that OPD's efficiency stems from a form of foresight'': it establishes a stable update trajectory toward the final model early in training.<n>We propose textbfEffOPD, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction.
  arXiv  Detail & Related papers  (2026-05-12T08:19:15Z)
• HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Driving [51.268878540511054]
  We propose HAD, an end-to-end planning framework with a Hierarchical Diffusion Policy.<n>We show that HAD achieves new state-of-the-art performance on both NAVSIM and HUGSIM.
  arXiv  Detail & Related papers  (2026-04-04T04:12:47Z)
• Guided by Trajectories: Repairing and Rewarding Tool-Use Trajectories for Tool-Integrated Reasoning [65.10602992874787]
  AutoTraj is a framework that automatically learns TIR by repairing and rewarding tool-use trajectories.<n> Experiments on real-world benchmarks demonstrate the effectiveness of AutoTraj.
  arXiv  Detail & Related papers  (2026-01-30T14:42:04Z)
• In-Context Reinforcement Learning From Suboptimal Historical Data [56.60512975858003]
  Transformer models have achieved remarkable empirical successes, largely due to their in-context learning capabilities.<n>We propose the Decision Importance Transformer framework, which emulates the actor-critic algorithm in an in-context manner.<n>Our results show that DIT achieves superior performance, particularly when the offline dataset contains suboptimal historical data.
  arXiv  Detail & Related papers  (2026-01-27T23:13:06Z)
• Decoupling Return-to-Go for Efficient Decision Transformer [6.429850804144503]
  Decision Transformer (DT) has established a powerful sequence modeling approach to offline reinforcement learning.<n>Decoupled DT (DDT) simplifies the architecture by processing only observation and action sequences through the Transformer.<n>Our experiments show that DDT significantly outperforms DT and establishes competitive performance against state-of-the-art DT variants.
  arXiv  Detail & Related papers  (2026-01-22T13:42:08Z)
• Diffusion-Sharpening: Fine-tuning Diffusion Models with Denoising Trajectory Sharpening [56.99266993852532]
  Diffusion-Sharpening is a fine-tuning approach that enhances downstream alignment by optimizing sampling trajectories. Our method demonstrates superior training efficiency with faster convergence, and best inference efficiency without requiring additional NFEs.
  arXiv  Detail & Related papers  (2025-02-17T18:57:26Z)
• Fast T2T: Optimization Consistency Speeds Up Diffusion-Based Training-to-Testing Solving for Combinatorial Optimization [83.65278205301576]
  We propose to learn direct mappings from different noise levels to the optimal solution for a given instance, facilitating high-quality generation with minimal shots. This is achieved through an optimization consistency training protocol, which minimizes the difference among samples. Experiments on two popular tasks, the Traveling Salesman Problem (TSP) and Maximal Independent Set (MIS), demonstrate the superiority of Fast T2T regarding both solution quality and efficiency.
  arXiv  Detail & Related papers  (2025-02-05T07:13:43Z)
• DRDT3: Diffusion-Refined Decision Test-Time Training Model [6.907105812732423]
  Decision Transformer (DT) has shown competitive performance compared to traditional offline reinforcement learning (RL) approaches. We introduce a unified framework, called Diffusion-Refined Decision TTT (DRDT3), to achieve performance beyond DT models.
  arXiv  Detail & Related papers  (2025-01-12T04:59:49Z)
• Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generation [29.952637757286073]
  Decision Transformer (DT) can learn effective policy from offline datasets by converting the offline reinforcement learning (RL) into a supervised sequence modeling task. We introduce Diffusion-Based Trajectory Branch Generation (BG), which expands the trajectories of the dataset with branches generated by a diffusion model. BG outperforms state-of-the-art sequence modeling methods on D4RL benchmark.
  arXiv  Detail & Related papers  (2024-11-18T06:44:14Z)
• Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers [111.78179839856293]
  Decision Transformers have emerged as a compelling paradigm for offline Reinforcement Learning (RL) Online finetuning of decision transformers has been surprisingly under-explored. We find that simply adding TD3 gradients to the finetuning process of ODT effectively improves the online finetuning performance of ODT.
  arXiv  Detail & Related papers  (2024-10-31T16:38:51Z)
• Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning [5.398202201395825]
  Decision Transformer (DT) has demonstrated exceptional capabilities in offline reinforcement learning. Decision ConvFormer (DC) is easier to understand in the context of modeling RL trajectories within a Markov Decision Process. We propose the Q-value Regularized Decision ConvFormer (QDC), which combines the understanding of RL trajectories by DC and incorporates a term that maximizes action values.
  arXiv  Detail & Related papers  (2024-09-12T14:10:22Z)
• Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation [67.13876021157887]
  Dynamic Tuning (DyT) is a novel approach to improve both parameter and inference efficiency for ViT adaptation. DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.
  arXiv  Detail & Related papers  (2024-03-18T14:05:52Z)
• DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning [19.84386060857712]
  This paper introduces DiffTORI, which utilizes Differentiable Trajectory optimization as the policy representation to generate actions for deep Reinforcement and Imitation learning. Across 15 model-based RL tasks and 35 imitation learning tasks with high-dimensional image and point cloud inputs, DiffTORI outperforms prior state-of-the-art methods in both domains.
  arXiv  Detail & Related papers  (2024-02-08T05:26:40Z)
• Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference [53.419249906014194]
  We study generative modeling for planning with datasets repurposed from offline reinforcement learning. We introduce the Latent Plan Transformer (), a novel model that leverages a latent variable to connect a Transformer-based trajectory generator and the final return.
  arXiv  Detail & Related papers  (2024-02-07T08:18:09Z)
• Context-Former: Stitching via Latent Conditioned Sequence Modeling [31.250234478757665]
  We introduce ContextFormer, which integrates contextual information-based imitation learning (IL) and sequence modeling to stitch sub-optimal trajectories. Experiments show ContextFormer can achieve competitive performance in multiple IL settings.
  arXiv  Detail & Related papers  (2024-01-29T06:05:14Z)
• Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets [30.044393664203483]
  We present a novel approach to enhance RvS methods by integrating intermediate targets. We introduce the Waypoint Transformer (WT), using an architecture that builds upon the DT framework and conditioned on automatically-generated waypoints. The results show a significant increase in the final return compared to existing RvS methods, with performance on par or greater than existing state-of-the-art temporal difference learning-based methods.
  arXiv  Detail & Related papers  (2023-06-24T22:25:29Z)
• Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuning [95.72622659619445]
  Delta tuning (DET) is deemed as the new paradigm for using pre-trained language models (PLMs) Up to now, various DETs with distinct design elements have been proposed, achieving performance on par with fine-tuning.
  arXiv  Detail & Related papers  (2022-10-24T14:57:35Z)
• Feasible Low-thrust Trajectory Identification via a Deep Neural Network Classifier [1.5076964620370268]
  This work proposes a deep neural network (DNN) to accurately identify feasible low thrust transfer prior to the optimization process. The DNN-classifier achieves an overall accuracy of 97.9%, which has the best performance among the tested algorithms.
  arXiv  Detail & Related papers  (2022-02-10T11:34:37Z)
• Event-Based Feature Tracking in Continuous Time with Sliding Window Optimization [55.11913183006984]
  We propose a novel method for continuous-time feature tracking in event cameras. We track features by aligning events along an estimated trajectory in space-time. We experimentally confirm that the proposed sliding-window B-spline optimization leads to longer and more accurate feature tracks.
  arXiv  Detail & Related papers  (2021-07-09T16:41:20Z)
• Dynamic Scale Training for Object Detection [111.33112051962514]
  We propose a Dynamic Scale Training paradigm (abbreviated as DST) to mitigate scale variation challenge in object detection. Experimental results demonstrate the efficacy of our proposed DST towards scale variation handling. It does not introduce inference overhead and could serve as a free lunch for general detection configurations.
  arXiv  Detail & Related papers  (2020-04-26T16:48:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.