Policy Architectures for Compositional Generalization in Control

• URL: http://arxiv.org/abs/2203.05960v1
• Date: Thu, 10 Mar 2022 06:44:24 GMT
• Title: Policy Architectures for Compositional Generalization in Control
• Authors: Allan Zhou, Vikash Kumar, Chelsea Finn, Aravind Rajeswaran
• Abstract summary: We introduce a framework for modeling entity-based compositional structure in tasks. Our policies are flexible and can be trained end-to-end without requiring any action primitives.
• Score: 71.61675703776628
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many tasks in control, robotics, and planning can be specified using desired goal configurations for various entities in the environment. Learning goal-conditioned policies is a natural paradigm to solve such tasks. However, current approaches struggle to learn and generalize as task complexity increases, such as variations in number of environment entities or compositions of goals. In this work, we introduce a framework for modeling entity-based compositional structure in tasks, and create suitable policy designs that can leverage this structure. Our policies, which utilize architectures like Deep Sets and Self Attention, are flexible and can be trained end-to-end without requiring any action primitives. When trained using standard reinforcement and imitation learning methods on a suite of simulated robot manipulation tasks, we find that these architectures achieve significantly higher success rates with less data. We also find these architectures enable broader and compositional generalization, producing policies that extrapolate to different numbers of entities than seen in training, and stitch together (i.e. compose) learned skills in novel ways. Videos of the results can be found at https://sites.google.com/view/comp-gen-rl.

Related papers

• Flex: End-to-End Text-Instructed Visual Navigation with Foundation Models [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies. Our findings are synthesized in Flex (Fly-lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors. We demonstrate the effectiveness of this approach on quadrotor fly-to-target tasks, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• Task Agnostic Architecture for Algorithm Induction via Implicit Composition [10.627575117586417]
  This position paper aims to explore developing such a unified architecture and proposes a theoretical framework of how it could be constructed. Recent Generative AI, especially Transformer-based models, demonstrate potential as an architecture capable of constructing algorithms for a wide range of domains. Our exploration delves into current capabilities and limitations of Transformer-based and other methods in efficient and correct algorithm composition.
  arXiv  Detail & Related papers  (2024-04-03T04:31:09Z)
• Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planning [2.7195102129095003]
  We propose a deep learning approach to perform multi-level object rearrangement planning for scenes with structural dependency hierarchies. It is trained on a self-generated simulation data set with intuitive structures and works for unseen scenes with an arbitrary number of objects. We compare our method with a range of classical and model-based baselines to show that our method leverages its scene understanding to achieve better performance, flexibility, and efficiency.
  arXiv  Detail & Related papers  (2023-09-05T19:35:44Z)
• Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space [76.46113138484947]
  General-purpose robots require diverse repertoires of behaviors to complete challenging tasks in real-world unstructured environments. To address this issue, goal-conditioned reinforcement learning aims to acquire policies that can reach goals for a wide range of tasks on command. We propose Planning to Practice, a method that makes it practical to train goal-conditioned policies for long-horizon tasks.
  arXiv  Detail & Related papers  (2022-05-17T06:58:17Z)
• Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates [63.58053355357644]
  We study the problem of learning a good set of policies, so that when combined together, they can solve a wide variety of unseen reinforcement learning tasks. We show theoretically that having access to a specific set of diverse policies, which we call a set of independent policies, can allow for instantaneously achieving high-level performance.
  arXiv  Detail & Related papers  (2021-12-30T12:20:46Z)
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals [20.29444813790076]
  Self-supervised agents set their own goals by exploiting the structure in the environment. Some of them were applied to learn basic manipulation skills in compositional multi-object environments. We propose a novel self-supervised agent that estimates relations between environment components and uses them to independently control different parts of the environment state.
  arXiv  Detail & Related papers  (2021-09-09T10:21:02Z)
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning [138.40338621974954]
  CausalWorld is a benchmark for causal structure and transfer learning in a robotic manipulation environment. Tasks consist of constructing 3D shapes from a given set of blocks - inspired by how children learn to build complex structures.
  arXiv  Detail & Related papers  (2020-10-08T23:01:13Z)
• MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architectures [61.73533544385352]
  We propose a transferable perturbation, MetaPerturb, which is meta-learned to improve generalization performance on unseen data. As MetaPerturb is a set-function trained over diverse distributions across layers and tasks, it can generalize heterogeneous tasks and architectures.
  arXiv  Detail & Related papers  (2020-06-13T02:54:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.