On Context Distribution Shift in Task Representation Learning for Offline Meta RL

• URL: http://arxiv.org/abs/2304.00354v2
• Date: Tue, 23 May 2023 13:14:57 GMT
• Title: On Context Distribution Shift in Task Representation Learning for Offline Meta RL
• Authors: Chenyang Zhao, Zihao Zhou, Bin Liu
• Abstract summary: We focus on context-based OMRL, specifically on the challenge of learning task representation for OMRL. To overcome this problem, we present a hard-sampling-based strategy to train a robust task context encoder.
• Score: 7.8317653074640186
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Offline Meta Reinforcement Learning (OMRL) aims to learn transferable knowledge from offline datasets to enhance the learning process for new target tasks. Context-based Reinforcement Learning (RL) adopts a context encoder to expediently adapt the agent to new tasks by inferring the task representation, and then adjusting the policy based on this inferred representation. In this work, we focus on context-based OMRL, specifically on the challenge of learning task representation for OMRL. We conduct experiments that demonstrate that the context encoder trained on offline datasets might encounter distribution shift between the contexts used for training and testing. To overcome this problem, we present a hard-sampling-based strategy to train a robust task context encoder. Our experimental findings on diverse continuous control tasks reveal that utilizing our approach yields more robust task representations and better testing performance in terms of accumulated returns compared to baseline methods. Our code is available at https://github.com/ZJLAB-AMMI/HS-OMRL.

Related papers

• CausalCOMRL: Context-Based Offline Meta-Reinforcement Learning with Causal Representation [13.575628222213387]
  CausalCOMRL is a context-based OMRL method that integrates causal representation learning. We show that CausalCOMRL achieves better performance than other methods on most benchmarks.
  arXiv  Detail & Related papers  (2025-02-03T01:43:54Z)
• Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning [12.443661471796595]
  offline meta-reinforcement learning aims to equip agents with the ability to rapidly adapt to new tasks by training on data from a set of different tasks. Context-based approaches utilize a history of state-action-reward transitions to infer representations of the current task, and then condition the agent, i.e., the policy and value function, on the task representations. Unfortunately, context-based approaches suffer from distribution mismatch, as the context in the offline data does not match the context at test time.
  arXiv  Detail & Related papers  (2024-12-19T13:24:01Z)
• Offline Reinforcement Learning from Datasets with Structured Non-Stationarity [50.35634234137108]
  Current Reinforcement Learning (RL) is often limited by the large amount of data needed to learn a successful policy. We address a novel Offline RL problem setting in which, while collecting the dataset, the transition and reward functions gradually change between episodes but stay constant within each episode. We propose a method based on Contrastive Predictive Coding that identifies this non-stationarity in the offline dataset, accounts for it when training a policy, and predicts it during evaluation.
  arXiv  Detail & Related papers  (2024-05-23T02:41:36Z)
• Offline Multitask Representation Learning for Reinforcement Learning [86.26066704016056]
  We study offline multitask representation learning in reinforcement learning (RL) We propose a new algorithm called MORL for offline multitask representation learning. Our theoretical results demonstrate the benefits of using the learned representation from the upstream offline task instead of directly learning the representation of the low-rank model.
  arXiv  Detail & Related papers  (2024-03-18T08:50:30Z)
• Hypernetworks for Zero-shot Transfer in Reinforcement Learning [21.994654567458017]
  Hypernetworks are trained to generate behaviors across a range of unseen task conditions. This work relates to meta RL, contextual RL, and transfer learning. Our method demonstrates significant improvements over baselines from multitask and meta RL approaches.
  arXiv  Detail & Related papers  (2022-11-28T15:48:35Z)
• Learning Action Translator for Meta Reinforcement Learning on Sparse-Reward Tasks [56.63855534940827]
  This work introduces a novel objective function to learn an action translator among training tasks. We theoretically verify that the value of the transferred policy with the action translator can be close to the value of the source policy. We propose to combine the action translator with context-based meta-RL algorithms for better data collection and more efficient exploration during meta-training.
  arXiv  Detail & Related papers  (2022-07-19T04:58:06Z)
• Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning [21.59254848913971]
  offline meta-reinforcement learning is a reinforcement learning paradigm that learns from offline data to adapt to new tasks. We propose a contrastive learning framework for task representations that are robust to the distribution of behavior policies in training and test. Experiments on a variety of offline meta-reinforcement learning benchmarks demonstrate the advantages of our method over prior methods.
  arXiv  Detail & Related papers  (2022-06-21T14:46:47Z)
• Provable Benefits of Representational Transfer in Reinforcement Learning [59.712501044999875]
  We study the problem of representational transfer in RL, where an agent first pretrains in a number of source tasks to discover a shared representation. We show that given generative access to source tasks, we can discover a representation, using which subsequent linear RL techniques quickly converge to a near-optimal policy.
  arXiv  Detail & Related papers  (2022-05-29T04:31:29Z)
• Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation [87.1188556802942]
  We present an approach for encoding visual task relationships to improve model performance in an Unsupervised Domain Adaptation (UDA) setting. We propose a novel Cross-Task Relation Layer (CTRL), which encodes task dependencies between the semantic and depth predictions. Furthermore, we propose an Iterative Self-Learning (ISL) training scheme, which exploits semantic pseudo-labels to provide extra supervision on the target domain.
  arXiv  Detail & Related papers  (2021-05-17T13:42:09Z)
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies [57.27944046925876]
  We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter.
  arXiv  Detail & Related papers  (2020-01-01T17:34:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.