Wasserstein Barycenter Soft Actor-Critic

• URL: http://arxiv.org/abs/2506.10167v2
• Date: Sat, 14 Jun 2025 21:29:34 GMT
• Title: Wasserstein Barycenter Soft Actor-Critic
• Authors: Zahra Shahrooei, Ali Baheri,
• Abstract summary: We propose Wasserstein Barycenter Soft Actor-Critic (WBSAC) algorithm, which benefits from a pessimistic actor for temporal difference learning and an optimistic actor to promote exploration.<n>We show that WBSAC is more sample-efficient on MuJoCo continuous control tasks.
• Score: 4.14360329494344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep off-policy actor-critic algorithms have emerged as the leading framework for reinforcement learning in continuous control domains. However, most of these algorithms suffer from poor sample efficiency, especially in environments with sparse rewards. In this paper, we take a step towards addressing this issue by providing a principled directed exploration strategy. We propose Wasserstein Barycenter Soft Actor-Critic (WBSAC) algorithm, which benefits from a pessimistic actor for temporal difference learning and an optimistic actor to promote exploration. This is achieved by using the Wasserstein barycenter of the pessimistic and optimistic policies as the exploration policy and adjusting the degree of exploration throughout the learning process. We compare WBSAC with state-of-the-art off-policy actor-critic algorithms and show that WBSAC is more sample-efficient on MuJoCo continuous control tasks.

Related papers

• IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic [52.44637913176449]
  This paper introduces the SOAR framework for imitation learning.<n>It is an algorithmic template that learns a policy from expert demonstrations with a primal dual style algorithm that alternates cost and policy updates.<n>It is shown to boost consistently the performance of imitation learning algorithms based on Soft Actor Critic such as f-IRL, ML-IRL and CSIL in several MuJoCo environments.
  arXiv  Detail & Related papers  (2025-02-27T08:03:37Z)
• PAC-Bayesian Soft Actor-Critic Learning [9.752336113724928]
  Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm.
  arXiv  Detail & Related papers  (2023-01-30T10:44:15Z)
• Bayesian Soft Actor-Critic: A Directed Acyclic Strategy Graph Based Deep Reinforcement Learning [1.8220718426493654]
  This paper proposes a novel directed acyclic strategy graph decomposition approach based on Bayesian chaining. We integrate this approach into the state-of-the-art DRL method -- soft actor-critic (SAC) We build the corresponding Bayesian soft actor-critic (BSAC) model by organizing several sub-policies as a joint policy.
  arXiv  Detail & Related papers  (2022-08-11T20:36:23Z)
• Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies [64.2210390071609]
  We present a novel Heavy-Tailed Policy Gradient (HT-PSG) algorithm to deal with the challenges of sparse rewards in continuous control problems. We show consistent performance improvement across all tasks in terms of high average cumulative reward.
  arXiv  Detail & Related papers  (2022-06-12T04:09:39Z)
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction [73.77593805292194]
  We train a separate exploration policy to maximize an approximate upper confidence bound of the critics in an off-policy actor-critic framework. To mitigate the off-policy-ness, we adapt the recently introduced DICE framework to learn a distribution correction ratio for off-policy actor-critic training.
  arXiv  Detail & Related papers  (2021-10-22T22:07:51Z)
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning [85.50033812217254]
  Actor-critic methods are widely used in offline reinforcement learning practice, but are not so well-understood theoretically. We propose a new offline actor-critic algorithm that naturally incorporates the pessimism principle.
  arXiv  Detail & Related papers  (2021-08-19T17:27:29Z)
• Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning [0.0]
  We propose Behavior-Guided Actor-Critic (BAC) as an off-policy actor-critic deep RL algorithm. BAC mathematically formulates the behavior of the policy through autoencoders. Results show considerably better performances of BAC when compared to several cutting-edge learning algorithms.
  arXiv  Detail & Related papers  (2021-04-09T15:22:35Z)
• OPAC: Opportunistic Actor-Critic [0.0]
  Opportunistic Actor-Critic (OPAC) is a novel model-free deep RL algorithm that employs better exploration policy and lesser variance. OPAC combines some of the most powerful features of TD3 and SAC and aims to optimize a policy in an off-policy way.
  arXiv  Detail & Related papers  (2020-12-11T18:33:35Z)
• Reannealing of Decaying Exploration Based On Heuristic Measure in Deep Q-Network [82.20059754270302]
  We propose an algorithm based on the idea of reannealing, that aims at encouraging exploration only when it is needed. We perform an illustrative case study showing that it has potential to both accelerate training and obtain a better policy.
  arXiv  Detail & Related papers  (2020-09-29T20:40:00Z)
• DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning [95.60782037764928]
  We show that simple Deterministic Policy Gradient works remarkably well as long as the overestimation bias is controlled. Second, we pinpoint training instabilities, typical of off-policy algorithms, to the greedy policy update step. Third, we show that ideas in the propensity estimation literature can be used to importance-sample transitions from replay buffer and update policy to prevent deterioration of performance.
  arXiv  Detail & Related papers  (2020-06-26T20:21:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.