Safe Reinforcement Learning for Legged Locomotion

• URL: http://arxiv.org/abs/2203.02638v1
• Date: Sat, 5 Mar 2022 01:49:16 GMT
• Title: Safe Reinforcement Learning for Legged Locomotion
• Abstract summary: We propose a safe reinforcement learning framework that switches between a safe recovery policy that prevents the robot from entering unsafe states, and a learner policy that is optimized to complete the task. We verify the proposed framework in four locomotion tasks on a simulated and real quadrupedal robot. Our method achieves 48.6% fewer falls and comparable or better rewards than the baseline methods in simulation.
• Score: 18.57681606310315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Designing control policies for legged locomotion is complex due to the under-actuated and non-continuous robot dynamics. Model-free reinforcement learning provides promising tools to tackle this challenge. However, a major bottleneck of applying model-free reinforcement learning in real world is safety. In this paper, we propose a safe reinforcement learning framework that switches between a safe recovery policy that prevents the robot from entering unsafe states, and a learner policy that is optimized to complete the task. The safe recovery policy takes over the control when the learner policy violates safety constraints, and hands over the control back when there are no future safety violations. We design the safe recovery policy so that it ensures safety of legged locomotion while minimally intervening in the learning process. Furthermore, we theoretically analyze the proposed framework and provide an upper bound on the task performance. We verify the proposed framework in four locomotion tasks on a simulated and real quadrupedal robot: efficient gait, catwalk, two-leg balance, and pacing. On average, our method achieves 48.6% fewer falls and comparable or better rewards than the baseline methods in simulation. When deployed it on real-world quadruped robot, our training pipeline enables 34% improvement in energy efficiency for the efficient gait, 40.9% narrower of the feet placement in the catwalk, and two times more jumping duration in the two-leg balance. Our method achieves less than five falls over the duration of 115 minutes of hardware time.

Related papers

• SLowRL: Safe Low-Rank Adaptation Reinforcement Learning for Locomotion [16.763723249971793]
  We introduce SLowRL, a framework that combines Low-Rank Adaptation (LoRA) with training-time safety enforcement via a recovery policy.<n> Experimental results show that our method achieves a $46.5%$ reduction in fine-tuning time and near-zero safety violations.
  arXiv  Detail & Related papers  (2026-03-17T19:26:48Z)
• Flow Policy Gradients for Robot Control [67.61978635211048]
  Flow matching policy gradients can be made effective for training and fine-tuning more expressive policies.<n>We show how policies can exploit the flow representation for exploration when training from scratch, as well as improved fine-tuning robustness over baselines.
  arXiv  Detail & Related papers  (2026-02-02T18:56:49Z)
• Dynamic Obstacle Avoidance with Bounded Rationality Adversarial Reinforcement Learning [5.760394464143113]
  We propose a novel way to endow navigation policies with robustness by a training process that models obstacles as adversarial agents. We call this method versa policies via Quantal response Adrial Reinforcement Learning (Hi-QARL)
  arXiv  Detail & Related papers  (2025-03-14T14:54:02Z)
• RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashes [57.319845580050924]
  We propose a reinforcement learning framework that combines risk-sensitive control with an adaptive action space curriculum. We show that our algorithm is capable of learning high-speed policies for a real-world off-road driving task.
  arXiv  Detail & Related papers  (2024-05-07T23:32:36Z)
• Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion [13.647294304606316]
  This paper introduces Agile But Safe (ABS), a learning-based control framework for quadrupedal robots. ABS involves an agile policy to execute agile motor skills amidst obstacles and a recovery policy to prevent failures. The training process involves the learning of the agile policy, the reach-avoid value network, the recovery policy, and an exteroception representation network.
  arXiv  Detail & Related papers  (2024-01-31T03:58:28Z)
• Towards Optimal Head-to-head Autonomous Racing with Curriculum Reinforcement Learning [22.69532642800264]
  We propose a head-to-head racing environment for reinforcement learning which accurately models vehicle dynamics. We also propose a control barrier function-based safe reinforcement learning algorithm to enforce the safety of the agent.
  arXiv  Detail & Related papers  (2023-08-25T17:05:41Z)
• Evaluation of Safety Constraints in Autonomous Navigation with Deep Reinforcement Learning [62.997667081978825]
  We compare two learnable navigation policies: safe and unsafe. The safe policy takes the constraints into the account, while the other does not. We show that the safe policy is able to generate trajectories with more clearance (distance to the obstacles) and makes less collisions while training without sacrificing the overall performance.
  arXiv  Detail & Related papers  (2023-07-27T01:04:57Z)
• Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios [147.16925581385576]
  We show how imitation learning combined with reinforcement learning can substantially improve the safety and reliability of driving policies. We train a policy on over 100k miles of urban driving data, and measure its effectiveness in test scenarios grouped by different levels of collision likelihood.
  arXiv  Detail & Related papers  (2022-12-21T23:59:33Z)
• Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning [64.11013095004786]
  We propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2022-12-14T03:11:25Z)
• Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages [42.57240271305088]
  We present a reinforcement learning based approach to autonomous vehicle longitudinal control, where the rule-based safety cages provide enhanced safety for the vehicle as well as weak supervision to the reinforcement learning agent. We show that when the model parameters are constrained or sub-optimal, the safety cages can enable a model to learn a safe driving policy even when the model could not be trained to drive through reinforcement learning alone.
  arXiv  Detail & Related papers  (2021-03-17T15:30:36Z)
• Protective Policy Transfer [37.897395735552706]
  We introduce a policy transfer algorithm for adapting robot motor skills to novel scenarios. Our algorithm trains two control policies: a task policy that is optimized to complete the task of interest, and a protective policy that is dedicated to keep the robot from unsafe events. We evaluate our approach on four simulated robot locomotion problems and a 2D navigation problem.
  arXiv  Detail & Related papers  (2020-12-11T22:10:54Z)
• Learning to be Safe: Deep RL with a Safety Critic [72.00568333130391]
  A natural first approach toward safe RL is to manually specify constraints on the policy's behavior. We propose to learn how to be safe in one set of tasks and environments, and then use that learned intuition to constrain future behaviors.
  arXiv  Detail & Related papers  (2020-10-27T20:53:20Z)
• Learning to Walk in the Real World with Minimal Human Effort [80.7342153519654]
  We develop a system for learning legged locomotion policies with deep RL in the real world with minimal human effort. Our system can automatically and efficiently learn locomotion skills on a Minitaur robot with little human intervention.
  arXiv  Detail & Related papers  (2020-02-20T03:36:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.