Latent Action Priors for Locomotion with Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2410.03246v2
• Date: Sat, 01 Mar 2025 09:12:55 GMT
• Title: Latent Action Priors for Locomotion with Deep Reinforcement Learning
• Authors: Oliver Hausdörfer, Alexander von Rohr, Éric Lefort, Angela Schoellig,
• Abstract summary: Deep Reinforcement Learning (DRL) enables robots to learn complex behaviors through interaction with the environment.<n>We propose an inductive bias for learning locomotion that is especially useful for torque control.<n>We observe that the agent is not restricted to the reward levels of the demonstration, and performance in transfer tasks is improved significantly.
• Score: 42.642008092347986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep Reinforcement Learning (DRL) enables robots to learn complex behaviors through interaction with the environment. However, due to the unrestricted nature of the learning algorithms, the resulting solutions are often brittle and appear unnatural. This is especially true for learning direct joint-level torque control, as inductive biases are difficult to integrate into the learning process. We propose an inductive bias for learning locomotion that is especially useful for torque control: latent actions learned from a small dataset of expert demonstrations. This prior allows the policy to directly leverage knowledge contained in the expert's actions and facilitates more efficient exploration. We observe that the agent is not restricted to the reward levels of the demonstration, and performance in transfer tasks is improved significantly. Latent action priors combined with style rewards for imitation lead to a closer replication of the expert's behavior. Videos and code are available at https://sites.google.com/view/latent-action-priors.

Related papers

• Pre-trained Visual Dynamics Representations for Efficient Policy Learning [33.62440075940917]
  We propose Pre-trained Visual Dynamics Representations (PVDR) to bridge the domain gap between videos and downstream tasks for efficient policy learning. The pre-trained visual dynamics representations capture the visual dynamics prior knowledge in the videos. This abstract prior knowledge can be readily adapted to downstream tasks and aligned with executable actions through online adaptation.
  arXiv  Detail & Related papers  (2024-11-05T15:18:02Z)
• RILe: Reinforced Imitation Learning [60.63173816209543]
  RILe (Reinforced Learning) is a framework that combines the strengths of imitation learning and inverse reinforcement learning to learn a dense reward function efficiently. Our framework produces high-performing policies in high-dimensional tasks where direct imitation fails to replicate complex behaviors.
  arXiv  Detail & Related papers  (2024-06-12T17:56:31Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery [22.32327908453603]
  Current reinforcement learning (RL) in robotics often experiences difficulty in generalizing to new downstream tasks. We propose a framework that pre-trains the agent in a task-agnostic manner without access to the task-specific reward. We show that our approach achieves the most diverse interacting behavior and significantly improves sample efficiency in downstream tasks.
  arXiv  Detail & Related papers  (2022-04-29T06:57:46Z)
• Action-Conditioned Contrastive Policy Pretraining [39.13710045468429]
  Deep visuomotor policy learning achieves promising results in control tasks such as robotic manipulation and autonomous driving. It requires a huge number of online interactions with the training environment, which limits its real-world application. In this work, we aim to pretrain policy representations for driving tasks using hours-long uncurated YouTube videos.
  arXiv  Detail & Related papers  (2022-04-05T17:58:22Z)
• Reinforcement Learning with Action-Free Pre-Training from Videos [95.25074614579646]
  We introduce a framework that learns representations useful for understanding the dynamics via generative pre-training on videos. Our framework significantly improves both final performances and sample-efficiency of vision-based reinforcement learning.
  arXiv  Detail & Related papers  (2022-03-25T19:44:09Z)
• Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning [7.51557557629519]
  We present Learning from Guided Play (LfGP), a framework in which we leverage expert demonstrations of, in addition to a main task, multiple auxiliary tasks. This affords many benefits: learning efficiency is improved for main tasks with challenging bottleneck transitions, expert data becomes reusable between tasks, and transfer learning through the reuse of learned auxiliary task models becomes possible.
  arXiv  Detail & Related papers  (2021-12-16T14:58:08Z)
• Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabeling [2.1485350418225244]
  Off-policy algorithms tend to be more sample-efficient, and can additionally benefit from any off-policy data stored in the replay buffer. Expert demonstrations are a popular source for such data. We present a new method, based on a reward bonus given to demonstrations and successful episodes.
  arXiv  Detail & Related papers  (2021-10-27T14:30:29Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• Demonstration-Guided Reinforcement Learning with Learned Skills [23.376115889936628]
  Demonstration-guided reinforcement learning (RL) is a promising approach for learning complex behaviors. In this work, we aim to exploit this shared subtask structure to increase the efficiency of demonstration-guided RL. We propose Skill-based Learning with Demonstrations (SkiLD), an algorithm for demonstration-guided RL that efficiently leverages the provided demonstrations.
  arXiv  Detail & Related papers  (2021-07-21T17:59:34Z)
• Hierarchical Few-Shot Imitation with Skill Transition Models [66.81252581083199]
  Few-shot Imitation with Skill Transition Models (FIST) is an algorithm that extracts skills from offline data and utilizes them to generalize to unseen tasks. We show that FIST is capable of generalizing to new tasks and substantially outperforms prior baselines in navigation experiments.
  arXiv  Detail & Related papers  (2021-07-19T15:56:01Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)
• Annotating Motion Primitives for Simplifying Action Search in Reinforcement Learning [10.764160559530847]
  Reinforcement learning in large-scale environments is challenging due to the many possible actions that can be taken in specific situations. We have previously developed a means of constraining, and hence speeding up, the search process through the use of motion primitives. We propose a theoretically viewpoint-insensitive and speed-insensitive means of automatically annotating the underlying motions and actions.
  arXiv  Detail & Related papers  (2021-02-24T01:32:06Z)
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning [79.32403825036792]
  We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials. We show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent's ability to try out novel behaviors.
  arXiv  Detail & Related papers  (2020-11-19T18:47:40Z)
• Learning Dexterous Manipulation from Suboptimal Experts [69.8017067648129]
  Relative Entropy Q-Learning (REQ) is a simple policy algorithm that combines ideas from successful offline and conventional RL algorithms. We show how REQ is also effective for general off-policy RL, offline RL, and RL from demonstrations.
  arXiv  Detail & Related papers  (2020-10-16T18:48:49Z)
• Bridging the Imitation Gap by Adaptive Insubordination [88.35564081175642]
  We show that when the teaching agent makes decisions with access to privileged information, this information is marginalized during imitation learning. We propose 'Adaptive Insubordination' (ADVISOR) to address this gap. ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration.
  arXiv  Detail & Related papers  (2020-07-23T17:59:57Z)
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting [66.45372974713189]
  We propose a recall and learn mechanism, which adopts the idea of multi-task learning and jointly learns pretraining tasks and downstream tasks. Experiments show that our method achieves state-of-the-art performance on the GLUE benchmark. We provide open-source RecAdam, which integrates the proposed mechanisms into Adam to facility the NLP community.
  arXiv  Detail & Related papers  (2020-04-27T08:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.