Distributional Reinforcement Learning with Regularized Wasserstein Loss

• URL: http://arxiv.org/abs/2202.00769v5
• Date: Mon, 14 Oct 2024 21:09:20 GMT
• Title: Distributional Reinforcement Learning with Regularized Wasserstein Loss
• Abstract summary: We propose textitSinkhorn distributional RL (SinkhornDRL), which leverages Sinkhorn divergence, a regularized Wasserstein loss, to minimize the difference between current and target Bellman return distributions. Empirically, we show that SinkhornDRL consistently outperforms or matches existing algorithms on the Atari games suite and particularly stands out in the multi-dimensional reward setting.
• Score: 16.042828467744197
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The empirical success of distributional reinforcement learning (RL) highly relies on the choice of distribution divergence equipped with an appropriate distribution representation. In this paper, we propose \textit{Sinkhorn distributional RL (SinkhornDRL)}, which leverages Sinkhorn divergence, a regularized Wasserstein loss, to minimize the difference between current and target Bellman return distributions. Theoretically, we prove the contraction properties of SinkhornDRL, aligning with the interpolation nature of Sinkhorn divergence between Wasserstein distance and Maximum Mean Discrepancy (MMD). The introduced SinkhornDRL enriches the family of distributional RL algorithms, contributing to interpreting the algorithm behaviors compared with existing approaches by our investigation into their relationships. Empirically, we show that SinkhornDRL consistently outperforms or matches existing algorithms on the Atari games suite and particularly stands out in the multi-dimensional reward setting. \thanks{Code is available in \url{https://github.com/datake/SinkhornDistRL}.}.

Related papers

• Multivariate Distributional Reinforcement Learning Using Sliced Divergences [37.5115685757579]
  Sliced Distributional Reinforcement Learning lifts tractable one-dimensional divergences to multivariate return distributions via projections.<n>We analyze Wasserstein, Cramér, and Maximum Mean Discrepancy (MMD)<n>We evaluate SDRL on a toy chain problem and a gridworld image-based environment as well as a subset of Atari games.
  arXiv  Detail & Related papers  (2026-05-29T12:26:45Z)
• Iterative Sampling Methods for Sinkhorn Distributionally Robust Optimization [2.345146665577353]
  Distributionally robust optimization (DRO) has emerged as a powerful paradigm for reliable decision-making under uncertainty.<n>This paper focuses on DRO with ambiguity sets defined via the Sinkhorn discrepancy: an entropy-regularized Wasserstein distance.<n>We study Sinkhorn DRO from the primal perspective, by reformulating it as a bilevel program with several infinite-dimensional lower-level subproblems over probability space.
  arXiv  Detail & Related papers  (2025-12-14T04:42:51Z)
• Uncertainty quantification for Markov chain induced martingales with application to temporal difference learning [55.197497603087065]
  We analyze the performance of the Temporal Difference (TD) learning algorithm with linear function approximations.<n>We establish novel and general high-dimensional concentration inequalities and Berry-Esseen bounds for vector-valued martingales induced by Markov chains.
  arXiv  Detail & Related papers  (2025-02-19T15:33:55Z)
• On the Power of Perturbation under Sampling in Solving Extensive-Form Games [56.013335390600524]
  We investigate how perturbation does and does not improve the Follow-the-Regularized-Leader (FTRL) algorithm in solving extensive-form games under sampling.<n>We present a unified framework for textitPerturbed FTRL algorithms and study two variants: PFTRL-KL and PFTRL-RKL.
  arXiv  Detail & Related papers  (2025-01-28T00:29:38Z)
• Reward-Directed Score-Based Diffusion Models via q-Learning [8.725446812770791]
  We propose a new reinforcement learning (RL) formulation for training continuous-time score-based diffusion models for generative AI. Our formulation does not involve any pretrained model for the unknown score functions of the noise-perturbed data distributions.
  arXiv  Detail & Related papers  (2024-09-07T13:55:45Z)
• More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning [58.626683114119906]
  We show that Distributional Reinforcement Learning (DistRL) can obtain second-order bounds in both online and offline RL. Our results are the first second-order bounds for low-rank MDPs and for offline RL.
  arXiv  Detail & Related papers  (2024-02-11T13:25:53Z)
• Adaptive Annealed Importance Sampling with Constant Rate Progress [68.8204255655161]
  Annealed Importance Sampling (AIS) synthesizes weighted samples from an intractable distribution. We propose the Constant Rate AIS algorithm and its efficient implementation for $alpha$-divergences.
  arXiv  Detail & Related papers  (2023-06-27T08:15:28Z)
• One-Step Distributional Reinforcement Learning [10.64435582017292]
  We present the simpler one-step distributional reinforcement learning (OS-DistrRL) framework. We show that our approach comes with a unified theory for both policy evaluation and control. We propose two OS-DistrRL algorithms for which we provide an almost sure convergence analysis.
  arXiv  Detail & Related papers  (2023-04-27T06:57:00Z)
• Policy Evaluation in Distributional LQR [70.63903506291383]
  We provide a closed-form expression of the distribution of the random return. We show that this distribution can be approximated by a finite number of random variables. Using the approximate return distribution, we propose a zeroth-order policy gradient algorithm for risk-averse LQR.
  arXiv  Detail & Related papers  (2023-03-23T20:27:40Z)
• Exponential Family Model-Based Reinforcement Learning via Score Matching [97.31477125728844]
  We propose an optimistic model-based algorithm, dubbed SMRL, for finitehorizon episodic reinforcement learning (RL) SMRL uses score matching, an unnormalized density estimation technique that enables efficient estimation of the model parameter by ridge regression.
  arXiv  Detail & Related papers  (2021-12-28T15:51:07Z)
• Distributional Reinforcement Learning for Multi-Dimensional Reward Functions [91.88969237680669]
  We introduce Multi-Dimensional Distributional DQN (MD3QN) to model the joint return distribution from multiple reward sources. As a by-product of joint distribution modeling, MD3QN can capture the randomness in returns for each source of reward. In experiments, our method accurately models the joint return distribution in environments with richly correlated reward functions.
  arXiv  Detail & Related papers  (2021-10-26T11:24:23Z)
• Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networks [7.907645828535088]
  The paper introduces a methodology for learning different representations of the random return distribution. A novel distributional RL algorithm named unconstrained monotonic deep Q-network (UMDQN) is presented.
  arXiv  Detail & Related papers  (2021-06-06T20:03:50Z)
• Forward and inverse reinforcement learning sharing network weights and hyperparameters [3.705785916791345]
  ERIL combines forward and inverse reinforcement learning (RL) under the framework of an entropy-regularized Markov decision process. A forward RL step minimizes the reverse KL estimated by the inverse RL step. We show that minimizing the reverse KL divergence is equivalent to finding an optimal policy.
  arXiv  Detail & Related papers  (2020-08-17T13:12:44Z)
• Nested-Wasserstein Self-Imitation Learning for Sequence Generation [158.19606942252284]
  We propose the concept of nested-Wasserstein distance for distributional semantic matching. A novel nested-Wasserstein self-imitation learning framework is developed, encouraging the model to exploit historical high-rewarded sequences.
  arXiv  Detail & Related papers  (2020-01-20T02:19:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.