Toward Task Generalization via Memory Augmentation in Meta-Reinforcement Learning

• URL: http://arxiv.org/abs/2502.01521v1
• Date: Mon, 03 Feb 2025 17:00:19 GMT
• Title: Toward Task Generalization via Memory Augmentation in Meta-Reinforcement Learning
• Authors: Kaixi Bao, Chenhao Li, Yarden As, Andreas Krause, Marco Hutter,
• Abstract summary: In reinforcement learning (RL), agents often struggle to perform well on tasks that differ from those encountered during training. This limitation presents a challenge to the broader deployment of RL in diverse and dynamic task settings. We introduce memory augmentation, a memory-based RL approach to improve task generalization.
• Score: 43.69919534800985
• License:
• Abstract: In reinforcement learning (RL), agents often struggle to perform well on tasks that differ from those encountered during training. This limitation presents a challenge to the broader deployment of RL in diverse and dynamic task settings. In this work, we introduce memory augmentation, a memory-based RL approach to improve task generalization. Our approach leverages task-structured augmentations to simulate plausible out-of-distribution scenarios and incorporates memory mechanisms to enable context-aware policy adaptation. Trained on a predefined set of tasks, our policy demonstrates the ability to generalize to unseen tasks through memory augmentation without requiring additional interactions with the environment. Through extensive simulation experiments and real-world hardware evaluations on legged locomotion tasks, we demonstrate that our approach achieves zero-shot generalization to unseen tasks while maintaining robust in-distribution performance and high sample efficiency.

Related papers

• Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Tasks [4.374837991804085]
  Task-Aware Virtual Training (TAVT) is a novel algorithm that captures task characteristics for both training and out-of-distribution (OOD) scenarios. Numerical results demonstrate that TAVT significantly enhances generalization to OOD tasks across various MuJoCo and MetaWorld environments.
  arXiv  Detail & Related papers  (2025-02-05T02:31:50Z)
• Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and Skills [17.666749042008178]
  We propose a framework called decoupled meta-reinforcement learning (DCMRL) DCMRL pulls in similar task contexts within the same task and pushing away different task contexts of different tasks. Experiments show that DCMRL is more effective than previous meta-RL methods with more generalizable prior experience.
  arXiv  Detail & Related papers  (2023-12-11T16:50:14Z)
• The Role of Diverse Replay for Generalisation in Reinforcement Learning [7.399291598113285]
  We investigate the impact of the exploration strategy and replay buffer on generalisation in reinforcement learning. We show that collecting and training on more diverse data from the training environments will improve zero-shot generalisation to new tasks.
  arXiv  Detail & Related papers  (2023-06-09T07:48:36Z)
• Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning [23.45043290237396]
  MoSS is a context-based Meta-reinforcement learning algorithm based on Self-Supervised task representation learning. On MuJoCo and Meta-World benchmarks, MoSS outperforms prior in terms of performance, sample efficiency (3-50x faster), adaptation efficiency, and generalization.
  arXiv  Detail & Related papers  (2023-04-29T15:46:19Z)
• Generalization in Visual Reinforcement Learning with the Reward Sequence Distribution [98.67737684075587]
  Generalization in partially observed markov decision processes (POMDPs) is critical for successful applications of visual reinforcement learning (VRL) We propose the reward sequence distribution conditioned on the starting observation and the predefined subsequent action sequence (RSD-OA) Experiments demonstrate that our representation learning approach based on RSD-OA significantly improves the generalization performance on unseen environments.
  arXiv  Detail & Related papers  (2023-02-19T15:47:24Z)
• Learning Action Translator for Meta Reinforcement Learning on Sparse-Reward Tasks [56.63855534940827]
  This work introduces a novel objective function to learn an action translator among training tasks. We theoretically verify that the value of the transferred policy with the action translator can be close to the value of the source policy. We propose to combine the action translator with context-based meta-RL algorithms for better data collection and more efficient exploration during meta-training.
  arXiv  Detail & Related papers  (2022-07-19T04:58:06Z)
• Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions [63.773813221460614]
  Generalization across different environments with the same tasks is critical for successful applications of visual reinforcement learning. We propose a novel approach, namely Characteristic Reward Sequence Prediction (CRESP), to extract the task-relevant information. Experiments demonstrate that CRESP significantly improves the performance of generalization on unseen environments.
  arXiv  Detail & Related papers  (2022-05-20T14:52:03Z)
• Multitask Adaptation by Retrospective Exploration with Learned World Models [77.34726150561087]
  We propose a meta-learned addressing model called RAMa that provides training samples for the MBRL agent taken from task-agnostic storage. The model is trained to maximize the expected agent's performance by selecting promising trajectories solving prior tasks from the storage.
  arXiv  Detail & Related papers  (2021-10-25T20:02:57Z)
• Meta-Reinforcement Learning in Broad and Non-Parametric Environments [8.091658684517103]
  We introduce TIGR, a Task-Inference-based meta-RL algorithm for tasks in non-parametric environments. We decouple the policy training from the task-inference learning and efficiently train the inference mechanism on the basis of an unsupervised reconstruction objective. We provide a benchmark with qualitatively distinct tasks based on the half-cheetah environment and demonstrate the superior performance of TIGR compared to state-of-the-art meta-RL approaches.
  arXiv  Detail & Related papers  (2021-08-08T19:32:44Z)
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning [90.93035276307239]
  We propose an information theoretic regularization objective and an annealing-based optimization method to achieve better generalization ability in RL agents. We demonstrate the extreme generalization benefits of our approach in different domains ranging from maze navigation to robotic tasks. This work provides a principled way to improve generalization in RL by gradually removing information that is redundant for task-solving.
  arXiv  Detail & Related papers  (2020-08-03T02:24:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.