Text-Aware Diffusion for Policy Learning

• URL: http://arxiv.org/abs/2407.01903v2
• Date: Thu, 31 Oct 2024 16:49:26 GMT
• Title: Text-Aware Diffusion for Policy Learning
• Authors: Calvin Luo, Mandy He, Zilai Zeng, Chen Sun,
• Abstract summary: We propose Text-Aware Diffusion for Policy Learning (TADPoLe), which uses a pretrained, frozen text-conditioned diffusion model to compute dense zero-shot reward signals for text-aligned policy learning. We show that TADPoLe is able to learn policies for novel goal-achievement and continuous locomotion behaviors specified by natural language, in both Humanoid and Dog environments.
• Score: 8.32790576855495
• License:
• Abstract: Training an agent to achieve particular goals or perform desired behaviors is often accomplished through reinforcement learning, especially in the absence of expert demonstrations. However, supporting novel goals or behaviors through reinforcement learning requires the ad-hoc design of appropriate reward functions, which quickly becomes intractable. To address this challenge, we propose Text-Aware Diffusion for Policy Learning (TADPoLe), which uses a pretrained, frozen text-conditioned diffusion model to compute dense zero-shot reward signals for text-aligned policy learning. We hypothesize that large-scale pretrained generative models encode rich priors that can supervise a policy to behave not only in a text-aligned manner, but also in alignment with a notion of naturalness summarized from internet-scale training data. In our experiments, we demonstrate that TADPoLe is able to learn policies for novel goal-achievement and continuous locomotion behaviors specified by natural language, in both Humanoid and Dog environments. The behaviors are learned zero-shot without ground-truth rewards or expert demonstrations, and are qualitatively more natural according to human evaluation. We further show that TADPoLe performs competitively when applied to robotic manipulation tasks in the Meta-World environment, without having access to any in-domain demonstrations.

Related papers

• Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control [73.6361029556484]
  Embodied AI agents require a fine-grained understanding of the physical world mediated through visual and language inputs. We consider pre-trained text-to-image diffusion models, which are explicitly optimized to generate images from text prompts. We show that Stable Control Representations enable learning policies that exhibit state-of-the-art performance on OVMM, a difficult open-vocabulary navigation benchmark.
  arXiv  Detail & Related papers  (2024-05-09T15:39:54Z)
• AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents [58.807802111818994]
  We propose AnySkill, a novel hierarchical method that learns physically plausible interactions following open-vocabulary instructions. Our approach begins by developing a set of atomic actions via a low-level controller trained via imitation learning. An important feature of our method is the use of image-based rewards for the high-level policy, which allows the agent to learn interactions with objects without manual reward engineering.
  arXiv  Detail & Related papers  (2024-03-19T15:41:39Z)
• Foundation Policies with Hilbert Representations [54.44869979017766]
  We propose an unsupervised framework to pre-train generalist policies from unlabeled offline data. Our key insight is to learn a structured representation that preserves the temporal structure of the underlying environment. Our experiments show that our unsupervised policies can solve goal-conditioned and general RL tasks in a zero-shot fashion.
  arXiv  Detail & Related papers  (2024-02-23T19:09:10Z)
• Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning [58.3994826169858]
  We introduce RoboFuME, a reset-free fine-tuning system for robotic reinforcement learning. Our insights are to utilize offline reinforcement learning techniques to ensure efficient online fine-tuning of a pre-trained policy. Our method can incorporate data from an existing robot dataset and improve on a target task within as little as 3 hours of autonomous real-world experience.
  arXiv  Detail & Related papers  (2023-10-23T17:50:08Z)
• Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments [45.213059639254475]
  We propose a new topic called imitator learning (ItorL) It aims to derive an imitator module that can reconstruct the imitation policies based on very limited expert demonstrations. For autonomous imitation policy building, we design a demonstration-based attention architecture for imitator policy.
  arXiv  Detail & Related papers  (2023-10-09T13:35:28Z)
• Learning Symbolic Rules over Abstract Meaning Representations for Textual Reinforcement Learning [63.148199057487226]
  We propose a modular, NEuroSymbolic Textual Agent (NESTA) that combines a generic semantic generalization with a rule induction system to learn interpretable rules as policies. Our experiments show that the proposed NESTA method outperforms deep reinforcement learning-based techniques by achieving better to unseen test games and learning from fewer training interactions.
  arXiv  Detail & Related papers  (2023-07-05T23:21:05Z)
• Contrastive Language, Action, and State Pre-training for Robot Learning [1.1000499414131326]
  We introduce a method for unifying language, action, and state information in a shared embedding space to facilitate a range of downstream tasks in robot learning. Our method, Contrastive Language, Action, and State Pre-training (CLASP), extends the CLIP formulation by incorporating distributional learning, capturing the inherent complexities and one-to-many relationships in behaviour-text alignment. We demonstrate the utility of our method for the following downstream tasks: zero-shot text-behaviour retrieval, captioning unseen robot behaviours, and learning a behaviour prior to language-conditioned reinforcement learning.
  arXiv  Detail & Related papers  (2023-04-21T07:19:33Z)
• Robust Learning from Observation with Model Misspecification [33.92371002674386]
  Imitation learning (IL) is a popular paradigm for training policies in robotic systems. We propose a robust IL algorithm to learn policies that can effectively transfer to the real environment without fine-tuning.
  arXiv  Detail & Related papers  (2022-02-12T07:04:06Z)
• Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning [75.56839075060819]
  Traditional robotic approaches rely on an accurate model of the environment, a detailed description of how to perform the task, and a robust perception system to keep track of the current state. reinforcement learning approaches can operate directly from raw sensory inputs with only a reward signal to describe the task, but are extremely sample-inefficient and brittle. In this work, we combine the strengths of model-based methods with the flexibility of learning-based methods to obtain a general method that is able to overcome inaccuracies in the robotics perception/actuation pipeline.
  arXiv  Detail & Related papers  (2020-05-21T19:47:05Z)
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text [12.88819706338837]
  Recent work has described neural-network-based agents that are trained with reinforcement learning to execute language-like commands in simulated worlds. We propose a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions.
  arXiv  Detail & Related papers  (2020-05-19T12:16:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.