Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint

• URL: http://arxiv.org/abs/2303.04356v2
• Date: Sun, 2 Jul 2023 08:48:56 GMT
• Title: Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint
• Abstract summary: Soft actor-critic (SAC) in reinforcement learning is expected to be one of the next-generation robot control schemes. Current SAC is no longer maximize the policy entropy, contrary to our expectation. This paper improves its implementation with a learnable state-dependent slack variable.
• Score: 8.071506311915396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Soft actor-critic (SAC) in reinforcement learning is expected to be one of the next-generation robot control schemes. Its ability to maximize policy entropy would make a robotic controller robust to noise and perturbation, which is useful for real-world robot applications. However, the priority of maximizing the policy entropy is automatically tuned in the current implementation, the rule of which can be interpreted as one for equality constraint, binding the policy entropy into its specified lower bound. The current SAC is therefore no longer maximize the policy entropy, contrary to our expectation. To resolve this issue in SAC, this paper improves its implementation with a learnable state-dependent slack variable for appropriately handling the inequality constraint to maximize the policy entropy by reformulating it as the corresponding equality constraint. The introduced slack variable is optimized by a switching-type loss function that takes into account the dual objectives of satisfying the equality constraint and checking the lower bound. In Mujoco and Pybullet simulators, the modified SAC statistically achieved the higher robustness for adversarial attacks than before while regularizing the norm of action. A real-robot variable impedance task was demonstrated for showing the applicability of the modified SAC to real-world robot control. In particular, the modified SAC maintained adaptive behaviors for physical human-robot interaction, which had no experience at all during training. https://youtu.be/EH3xVtlVaJw

Related papers

• ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback [1.711436622393758]
  Online Residual Policy Adaptation (ORPA) is a framework that enables feedback-driven correction of robot actions without modifying the underlying policy parameters.<n>ORPA augments a pretrained control policy with a lightweight, feedback-conditioned module that predicts residual adjustments directly in joint space.<n>We evaluate ORPA on a set of precision-sensitive manipulation tasks using the ALOHA platform, demonstrating improvements in success rate and recovery from small perturbations compared to baseline control policies and rule-based inverse kinematics corrections.
  arXiv  Detail & Related papers  (2026-08-18T03:30:12Z)
• PPO-EAL: Exact Augmented Lagrangian Proximal Policy Optimization for Safe Robotic Control [8.882504398064087]
  PPO-EAL is a novel first-order constrained policy optimization framework.<n>It integrates exact augmented Lagrangian optimization into policy optimization for safe robotic control.<n>It achieves theoretically grounded constraint enforcement without requiring impractically large penalty factors.
  arXiv  Detail & Related papers  (2026-06-26T08:59:38Z)
• Bridging the Gap: Enabling Soft Actor Critic for High Performance Legged Locomotion [14.693602704450528]
  Soft Actor-Critic is an off-policy algorithm that can reuse past experience.<n>It is a natural candidate for simto-real transfer where the same algorithm can be used both in simulation and for online learning on the real robot.<n>This work identifies the root causes of this gap and introduces targeted modifications that enable SAC to train stably at scale.
  arXiv  Detail & Related papers  (2026-05-24T10:04:11Z)
• FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control [55.38832429564216]
  Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable.<n>We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic.<n>Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency.
  arXiv  Detail & Related papers  (2026-04-06T09:03:41Z)
• REBEL: A Regularization-Based Solution for Reward Overoptimization in Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and user intentions, values, or social norms can be catastrophic in the real world. Current methods to mitigate this misalignment work by learning reward functions from human preferences. We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• Probabilistic Reach-Avoid for Bayesian Neural Networks [71.67052234622781]
  We show that an optimal synthesis algorithm can provide more than a four-fold increase in the number of certifiable states. The algorithm is able to provide more than a three-fold increase in the average guaranteed reach-avoid probability.
  arXiv  Detail & Related papers  (2023-10-03T10:52:21Z)
• Improved Policy Optimization for Online Imitation Learning [17.450401609682544]
  We consider online imitation learning (OIL), where the task is to find a policy that imitates the behavior of an expert via active interaction with the environment.
  arXiv  Detail & Related papers  (2022-07-29T22:02:14Z)
• Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization [63.75188254377202]
  Deep reinforcement learning algorithms can perform poorly in real-world tasks due to discrepancy between source and target environments. We propose a novel model-free actor-critic algorithm to learn robust policies without modeling the disturbance in advance. Experiments in several robot control tasks demonstrate that SCPO learns robust policies against the disturbance in transition dynamics.
  arXiv  Detail & Related papers  (2021-12-20T13:13:05Z)
• Teaching a Robot to Walk Using Reinforcement Learning [0.0]
  reinforcement learning can train optimal walking policies with ease. We teach a simulated two-dimensional bipedal robot how to walk using the OpenAI Gym BipedalWalker-v3 environment. ARS resulted in a better trained robot, and produced an optimal policy which officially "solves" the BipedalWalker-v3 problem.
  arXiv  Detail & Related papers  (2021-12-13T21:35:45Z)
• Continuous-Time Fitted Value Iteration for Robust Policies [93.25997466553929]
  Solving the Hamilton-Jacobi-Bellman equation is important in many domains including control, robotics and economics. We propose continuous fitted value iteration (cFVI) and robust fitted value iteration (rFVI) These algorithms leverage the non-linear control-affine dynamics and separable state and action reward of many continuous control problems.
  arXiv  Detail & Related papers  (2021-10-05T11:33:37Z)
• Robust Value Iteration for Continuous Control Tasks [99.00362538261972]
  When transferring a control policy from simulation to a physical system, the policy needs to be robust to variations in the dynamics to perform well. We present Robust Fitted Value Iteration, which uses dynamic programming to compute the optimal value function on the compact state domain. We show that robust value is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm.
  arXiv  Detail & Related papers  (2021-05-25T19:48:35Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.