GRAM: Generalization in Deep RL with a Robust Adaptation Module

• URL: http://arxiv.org/abs/2412.04323v1
• Date: Thu, 05 Dec 2024 16:39:01 GMT
• Title: GRAM: Generalization in Deep RL with a Robust Adaptation Module
• Authors: James Queeney, Xiaoyi Cai, Mouhacine Benosman, Jonathan P. How,
• Abstract summary: In this work, we present a framework for dynamics generalization in deep reinforcement learning. We introduce a robust adaptation module that provides a mechanism for identifying and reacting to both in-distribution and out-of-distribution environment dynamics. Our algorithm GRAM achieves strong generalization performance across in-distribution and out-of-distribution scenarios upon deployment.
• Score: 29.303051759538416
• License:
• Abstract: The reliable deployment of deep reinforcement learning in real-world settings requires the ability to generalize across a variety of conditions, including both in-distribution scenarios seen during training as well as novel out-of-distribution scenarios. In this work, we present a framework for dynamics generalization in deep reinforcement learning that unifies these two distinct types of generalization within a single architecture. We introduce a robust adaptation module that provides a mechanism for identifying and reacting to both in-distribution and out-of-distribution environment dynamics, along with a joint training pipeline that combines the goals of in-distribution adaptation and out-of-distribution robustness. Our algorithm GRAM achieves strong generalization performance across in-distribution and out-of-distribution scenarios upon deployment, which we demonstrate on a variety of realistic simulated locomotion tasks with a quadruped robot.

Related papers

• Toward Task Generalization via Memory Augmentation in Meta-Reinforcement Learning [43.69919534800985]
  In reinforcement learning (RL), agents often struggle to perform well on tasks that differ from those encountered during training. This limitation presents a challenge to the broader deployment of RL in diverse and dynamic task settings. We introduce memory augmentation, a memory-based RL approach to improve task generalization.
  arXiv  Detail & Related papers  (2025-02-03T17:00:19Z)
• Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity [10.402855891273346]
  DIVA is an evolutionary approach for generating diverse training tasks in complex, open-ended simulators. Our empirical results showcase DIVA's unique ability to overcome complex parameterizations and successfully train adaptive agent behavior.
  arXiv  Detail & Related papers  (2024-11-07T06:27:12Z)
• Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations [22.6449779859417]
  General intelligence requires quick adaption across tasks. In this paper, we explore a wider range of scenarios where not only the distribution but also the environment spaces may change. We introduce a causality-guided self-adaptive representation-based approach, called CSR, that equips the agent to generalize effectively.
  arXiv  Detail & Related papers  (2024-07-30T08:48:49Z)
• Distribution-Dependent Rates for Multi-Distribution Learning [26.38831409926518]
  Recent multi-distribution learning framework tackles this objective in a dynamic interaction with the environment. We provide distribution-dependent guarantees in the MDL regime, that scale with suboptimality gaps and result in superior dependence on the sample size. We devise an adaptive optimistic algorithm, LCB-DR, that showcases enhanced dependence on the gaps, mirroring the contrast between uniform and optimistic allocation in the multi-armed bandit literature.
  arXiv  Detail & Related papers  (2023-12-20T15:50:16Z)
• Distributionally Adaptive Meta Reinforcement Learning [85.17284589483536]
  We develop a framework for meta-RL algorithms that behave appropriately under test-time distribution shifts. Our framework centers on an adaptive approach to distributional robustness that trains a population of meta-policies to be robust to varying levels of distribution shift. We show how our framework allows for improved regret under distribution shift, and empirically show its efficacy on simulated robotics problems.
  arXiv  Detail & Related papers  (2022-10-06T17:55:09Z)
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning [137.39196753245105]
  We present a new model-based reinforcement learning algorithm that learns a multi-headed dynamics model for dynamics generalization. We incorporate context learning, which encodes dynamics-specific information from past experiences into the context latent vector. Our method exhibits superior zero-shot generalization performance across a variety of control tasks, compared to state-of-the-art RL methods.
  arXiv  Detail & Related papers  (2020-10-26T03:20:42Z)
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning [124.9856253431878]
  We decompose the task of learning a global dynamics model into two stages: (a) learning a context latent vector that captures the local dynamics, then (b) predicting the next state conditioned on it. In order to encode dynamics-specific information into the context latent vector, we introduce a novel loss function that encourages the context latent vector to be useful for predicting both forward and backward dynamics. The proposed method achieves superior generalization ability across various simulated robotics and control tasks, compared to existing RL schemes.
  arXiv  Detail & Related papers  (2020-05-14T08:10:54Z)
• From Simulation to Real World Maneuver Execution using Deep Reinforcement Learning [69.23334811890919]
  Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios. This is mainly due to the lack of domain adaptation between simulated and real-world data together with the absence of distinction between train and test datasets. We present a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios.
  arXiv  Detail & Related papers  (2020-05-13T14:22:20Z)
• Invariant Causal Prediction for Block MDPs [106.63346115341862]
  Generalization across environments is critical to the successful application of reinforcement learning algorithms to real-world challenges. We propose a method of invariant prediction to learn model-irrelevance state abstractions (MISA) that generalize to novel observations in the multi-environment setting.
  arXiv  Detail & Related papers  (2020-03-12T21:03:01Z)
• Target-Embedding Autoencoders for Supervised Representation Learning [111.07204912245841]
  This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional. We motivate and formalize the general framework of target-embedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets.
  arXiv  Detail & Related papers  (2020-01-23T02:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.