Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution

• URL: http://arxiv.org/abs/2309.12529v1
• Date: Thu, 21 Sep 2023 22:58:59 GMT
• Title: Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution
• Authors: Shuang Ao, Tianyi Zhou, Guodong Long, Xuan Song, Jing Jiang
• Abstract summary: We optimize an RL agent and its morphology through morphology-environment co-evolution'' Instead of hand-crafting the curriculum, we train two policies to automatically change the morphology and the environment.
• Score: 46.27211830466317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Throughout long history, natural species have learned to survive by evolving their physical structures adaptive to the environment changes. In contrast, current reinforcement learning (RL) studies mainly focus on training an agent with a fixed morphology (e.g., skeletal structure and joint attributes) in a fixed environment, which can hardly generalize to changing environments or new tasks. In this paper, we optimize an RL agent and its morphology through ``morphology-environment co-evolution (MECE)'', in which the morphology keeps being updated to adapt to the changing environment, while the environment is modified progressively to bring new challenges and stimulate the improvement of the morphology. This leads to a curriculum to train generalizable RL, whose morphology and policy are optimized for different environments. Instead of hand-crafting the curriculum, we train two policies to automatically change the morphology and the environment. To this end, (1) we develop two novel and effective rewards for the two policies, which are solely based on the learning dynamics of the RL agent; (2) we design a scheduler to automatically determine when to change the environment and the morphology. In experiments on two classes of tasks, the morphology and RL policies trained via MECE exhibit significantly better generalization performance in unseen test environments than SOTA morphology optimization methods. Our ablation studies on the two MECE policies further show that the co-evolution between the morphology and environment is the key to the success.

Related papers

• Learning to Predict Mutation Effects of Protein-Protein Interactions by Microenvironment-aware Hierarchical Prompt Learning [78.38442423223832]
  We develop a novel codebook pre-training task, namely masked microenvironment modeling. We demonstrate superior performance and training efficiency over state-of-the-art pre-training-based methods in mutation effect prediction.
  arXiv  Detail & Related papers  (2024-05-16T03:53:21Z)
• Evolving Reservoirs for Meta Reinforcement Learning [1.6874375111244329]
  We propose a computational model for studying a mechanism that can enable such a process. At the evolutionary scale, we evolve reservoirs, a family of recurrent neural networks. We employ these evolved reservoirs to facilitate the learning of a behavioral policy through Reinforcement Learning (RL) Our results show that the evolution of reservoirs can improve the learning of diverse challenging tasks.
  arXiv  Detail & Related papers  (2023-12-09T16:11:48Z)
• A Framework for Characterizing Novel Environment Transformations in General Environments [3.0938904602244346]
  We introduce a formal and theoretical framework for defining and categorizing environment transformations. We present a new language for describing domains, scenario generators, and transformations. We offer the first formal and computational set of tests for eight categories of environment transformations.
  arXiv  Detail & Related papers  (2023-05-07T15:53:07Z)
• Task-Agnostic Morphology Evolution [94.97384298872286]
  Current approaches that co-adapt morphology and behavior use a specific task's reward as a signal for morphology optimization. This often requires expensive policy optimization and results in task-dependent morphologies that are not built to generalize. We propose a new approach, Task-Agnostic Morphology Evolution (TAME), to alleviate both of these issues.
  arXiv  Detail & Related papers  (2021-02-25T18:59:21Z)
• Embodied Intelligence via Learning and Evolution [92.26791530545479]
  We show that environmental complexity fosters the evolution of morphological intelligence. We also show that evolution rapidly selects morphologies that learn faster. Our experiments suggest a mechanistic basis for both the Baldwin effect and the emergence of morphological intelligence.
  arXiv  Detail & Related papers  (2021-02-03T18:58:31Z)
• Emergent Hand Morphology and Control from Optimizing Robust Grasps of Diverse Objects [63.89096733478149]
  We introduce a data-driven approach where effective hand designs naturally emerge for the purpose of grasping diverse objects. We develop a novel Bayesian Optimization algorithm that efficiently co-designs the morphology and grasping skills jointly. We demonstrate the effectiveness of our approach in discovering robust and cost-efficient hand morphologies for grasping novel objects.
  arXiv  Detail & Related papers  (2020-12-22T17:52:29Z)
• Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments [11.076005074172516]
  We propose a systematic incremental learning method for Evolution strategies (ES) in dynamic environments. The goal is to adjust previously learned policy to a new one incrementally whenever the environment changes. This paper introduces a family of scalable ES algorithms for RL domains that enables rapid learning adaptation to dynamic environments.
  arXiv  Detail & Related papers  (2020-10-09T14:31:44Z)
• Ecological Reinforcement Learning [76.9893572776141]
  We study the kinds of environment properties that can make learning under such conditions easier. understanding how properties of the environment impact the performance of reinforcement learning agents can help us to structure our tasks in ways that make learning tractable.
  arXiv  Detail & Related papers  (2020-06-22T17:55:03Z)
• Environmental Adaptation of Robot Morphology and Control through Real-world Evolution [5.08706161686979]
  We apply evolutionary search to yield combinations of morphology and control for our mechanically self-reconfiguring quadruped robot. We evolve solutions on two distinct physical surfaces and analyze the results in terms of both control and morphology.
  arXiv  Detail & Related papers  (2020-03-30T07:57:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.