Distributional Bellman Operators over Mean Embeddings

• URL: http://arxiv.org/abs/2312.07358v3
• Date: Mon, 4 Mar 2024 16:54:22 GMT
• Title: Distributional Bellman Operators over Mean Embeddings
• Authors: Li Kevin Wenliang, Gr\'egoire Del\'etang, Matthew Aitchison, Marcus Hutter, Anian Ruoss, Arthur Gretton, Mark Rowland
• Abstract summary: We propose a novel framework for distributional reinforcement learning, based on learning finite-dimensional mean embeddings of return distributions. We derive several new algorithms for dynamic programming and temporal-difference learning based on this framework.
• Score: 37.5480897544168
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a novel algorithmic framework for distributional reinforcement learning, based on learning finite-dimensional mean embeddings of return distributions. We derive several new algorithms for dynamic programming and temporal-difference learning based on this framework, provide asymptotic convergence theory, and examine the empirical performance of the algorithms on a suite of tabular tasks. Further, we show that this approach can be straightforwardly combined with deep reinforcement learning, and obtain a new deep RL agent that improves over baseline distributional approaches on the Arcade Learning Environment.

Related papers

• GRAWA: Gradient-based Weighted Averaging for Distributed Training of Deep Learning Models [9.377424534371727]
  We study distributed training of deep models in time-constrained environments. We propose a new algorithm that periodically pulls workers towards the center variable computed as an average of workers.
  arXiv  Detail & Related papers  (2024-03-07T04:22:34Z)
• Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts [11.765000124617186]
  We study the robustness of deep reinforcement learning algorithms against distribution shifts within contextual multi-stage optimization problems. We show that our algorithm is superior to risk-neutral Soft Actor-Critic as well as to two benchmark approaches for robust deep reinforcement learning.
  arXiv  Detail & Related papers  (2024-02-15T14:55:38Z)
• A Distributional Analogue to the Successor Representation [54.99439648059807]
  This paper contributes a new approach for distributional reinforcement learning. It elucidates a clean separation of transition structure and reward in the learning process. As an illustration, we show that it enables zero-shot risk-sensitive policy evaluation.
  arXiv  Detail & Related papers  (2024-02-13T15:35:24Z)
• Towards a Systematic Approach to Design New Ensemble Learning Algorithms [0.0]
  This study revisits the foundational work on ensemble error decomposition. Recent advancements introduced a "unified theory of diversity" Our research systematically explores the application of this decomposition to guide the creation of new ensemble learning algorithms.
  arXiv  Detail & Related papers  (2024-02-09T22:59:20Z)
• Stochastic Unrolled Federated Learning [85.6993263983062]
  We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning. Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
  arXiv  Detail & Related papers  (2023-05-24T17:26:22Z)
• On the Convergence of Distributed Stochastic Bilevel Optimization Algorithms over a Network [55.56019538079826]
  Bilevel optimization has been applied to a wide variety of machine learning models. Most existing algorithms restrict their single-machine setting so that they are incapable of handling distributed data. We develop novel decentralized bilevel optimization algorithms based on a gradient tracking communication mechanism and two different gradients.
  arXiv  Detail & Related papers  (2022-06-30T05:29:52Z)
• Reinforcement Learning as Iterative and Amortised Inference [62.997667081978825]
  We use the control as inference framework to outline a novel classification scheme based on amortised and iterative inference. We show that taking this perspective allows us to identify parts of the algorithmic design space which have been relatively unexplored.
  arXiv  Detail & Related papers  (2020-06-13T16:10:03Z)
• A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms [67.67377846416106]
  We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes. We show that value-based methods such as TD($lambda$) and $Q$-Learning have update rules which are contractive in the space of distributions of functions.
  arXiv  Detail & Related papers  (2020-03-27T05:13:29Z)
• Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning [6.16852156844376]
  We describe a novel framework, Inferential Induction, for correctly inferring value function distributions from data. We experimentally demonstrate that the proposed algorithm is competitive with respect to the state of the art.
  arXiv  Detail & Related papers  (2020-02-08T06:19:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.