TEA: Trajectory Encoding Augmentation for Robust and Transferable Policies in Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2411.19133v2
• Date: Sun, 26 Jan 2025 22:16:57 GMT
• Title: TEA: Trajectory Encoding Augmentation for Robust and Transferable Policies in Offline Reinforcement Learning
• Authors: Batıkan Bora Ormancı, Phillip Swazinna, Steffen Udluft, Thomas A. Runkler,
• Abstract summary: We propose Trajectory Augmentation (TEA), which extends the state space by integrating latent representations of environmental dynamics obtained from sequence encoders.<n>Our findings show that incorporating these encodings with TEA improves the transferability of a single policy to novel environments.<n>These results indicate that TEA captures critical, environment-specific characteristics, enabling agents to generalize effectively across dynamic conditions.
• Score: 6.462260690750607
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we investigate offline reinforcement learning (RL) with the goal of training a single robust policy that generalizes effectively across environments with unseen dynamics. We propose a novel approach, Trajectory Encoding Augmentation (TEA), which extends the state space by integrating latent representations of environmental dynamics obtained from sequence encoders, such as AutoEncoders. Our findings show that incorporating these encodings with TEA improves the transferability of a single policy to novel environments with new dynamics, surpassing methods that rely solely on unmodified states. These results indicate that TEA captures critical, environment-specific characteristics, enabling RL agents to generalize effectively across dynamic conditions.

Related papers

• Exploration and Adaptation in Non-Stationary Tasks with Diffusion Policies [0.0]
  This paper investigates the application of Diffusion Policy in non-stationary, vision-based RL settings, specifically targeting environments where task dynamics and objectives evolve over time. We apply Diffusion Policy -- which leverages iterative denoising to refine latent action representations-to benchmark environments including Procgen and PointMaze. Our experiments demonstrate that, despite increased computational demands, Diffusion Policy consistently outperforms standard RL methods such as PPO and DQN, achieving higher mean and maximum rewards with reduced variability.
  arXiv  Detail & Related papers  (2025-03-31T23:00:07Z)
• Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning [53.9544543607396]
  We propose a novel framework that integrates reward rendering with Imitation from Observation (IfO) By instantiating F-distance in different ways, we derive two theoretical analysis and develop a practical algorithm called Accessible State Oriented Policy Regularization (ASOR) ASOR serves as a general add-on module that can be incorporated into various approaches RL, including offline RL and off-policy RL.
  arXiv  Detail & Related papers  (2025-03-10T03:50:20Z)
• Doubly-Dynamic ISAC Precoding for Vehicular Networks: A Constrained Deep Reinforcement Learning (CDRL) Approach [11.770137653756697]
  Integrated sensing and communication (ISAC) technology is essential for supporting vehicular networks. The communication channel in this scenario exhibits time variations, and the potential targets may move rapidly, resulting in double dynamics. We propose using constrained deep reinforcement learning to facilitate dynamic updates to the ISAC precoder.
  arXiv  Detail & Related papers  (2024-05-23T09:19:14Z)
• A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning [3.1515473193934778]
  Off-dynamics Reinforcement Learning seeks to transfer a policy from a source environment to a target environment characterized by distinct yet similar dynamics. We propose an innovative approach inspired by recent advancements in Imitation Learning and conservative RL algorithms.
  arXiv  Detail & Related papers  (2023-12-24T13:09:08Z)
• Rethinking Decision Transformer via Hierarchical Reinforcement Learning [54.3596066989024]
  Decision Transformer (DT) is an innovative algorithm leveraging recent advances of the transformer architecture in reinforcement learning (RL) We introduce a general sequence modeling framework for studying sequential decision making through the lens of Hierarchical RL. We show DT emerges as a special case of this framework with certain choices of high-level and low-level policies, and discuss the potential failure of these choices.
  arXiv  Detail & Related papers  (2023-11-01T03:32:13Z)
• Reparameterized Policy Learning for Multimodal Trajectory Optimization [61.13228961771765]
  We investigate the challenge of parametrizing policies for reinforcement learning in high-dimensional continuous action spaces. We propose a principled framework that models the continuous RL policy as a generative model of optimal trajectories. We present a practical model-based RL method, which leverages the multimodal policy parameterization and learned world model.
  arXiv  Detail & Related papers  (2023-07-20T09:05:46Z)
• Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization [29.61829620717385]
  Key challenge of continual reinforcement learning (CRL) in dynamic environments is to promptly adapt the RL agent's behavior as the environment changes over its lifetime. DaCoRL learns a context-conditioned policy using progressive contextualization. DaCoRL features consistent superiority over existing methods in terms of the stability, overall performance and generalization ability.
  arXiv  Detail & Related papers  (2022-09-01T10:26:58Z)
• A Regularized Implicit Policy for Offline Reinforcement Learning [54.7427227775581]
  offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. We propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. Experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.
  arXiv  Detail & Related papers  (2022-02-19T20:22:04Z)
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs [71.47895794305883]
  We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning setting. We present an SPI for this RL setting that takes into account the preferences of the algorithm's user for handling the trade-offs for different reward signals.
  arXiv  Detail & Related papers  (2021-05-31T21:04:21Z)
• Learning to Continuously Optimize Wireless Resource in a Dynamic Environment: A Bilevel Optimization Perspective [52.497514255040514]
  This work develops a new approach that enables data-driven methods to continuously learn and optimize resource allocation strategies in a dynamic environment. We propose to build the notion of continual learning into wireless system design, so that the learning model can incrementally adapt to the new episodes. Our design is based on a novel bilevel optimization formulation which ensures certain fairness" across different data samples.
  arXiv  Detail & Related papers  (2021-05-03T07:23:39Z)
• Robust Reinforcement Learning via Adversarial training with Langevin Dynamics [51.234482917047835]
  We introduce a sampling perspective to tackle the challenging task of training robust Reinforcement Learning (RL) agents. We present a novel, scalable two-player RL algorithm, which is a sampling variant of the two-player policy method.
  arXiv  Detail & Related papers  (2020-02-14T14:59:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.