A Distributional Analogue to the Successor Representation

• URL: http://arxiv.org/abs/2402.08530v2
• Date: Fri, 24 May 2024 16:29:32 GMT
• Title: A Distributional Analogue to the Successor Representation
• Authors: Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Yunhao Tang, André Barreto, Will Dabney, Marc G. Bellemare, Mark Rowland,
• Abstract summary: This paper contributes a new approach for distributional reinforcement learning. It elucidates a clean separation of transition structure and reward in the learning process. As an illustration, we show that it enables zero-shot risk-sensitive policy evaluation.
• Score: 54.99439648059807
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper contributes a new approach for distributional reinforcement learning which elucidates a clean separation of transition structure and reward in the learning process. Analogous to how the successor representation (SR) describes the expected consequences of behaving according to a given policy, our distributional successor measure (SM) describes the distributional consequences of this behaviour. We formulate the distributional SM as a distribution over distributions and provide theory connecting it with distributional and model-based reinforcement learning. Moreover, we propose an algorithm that learns the distributional SM from data by minimizing a two-level maximum mean discrepancy. Key to our method are a number of algorithmic techniques that are independently valuable for learning generative models of state. As an illustration of the usefulness of the distributional SM, we show that it enables zero-shot risk-sensitive policy evaluation in a way that was not previously possible.

Related papers

• Distribution-Dependent Rates for Multi-Distribution Learning [26.38831409926518]
  Recent multi-distribution learning framework tackles this objective in a dynamic interaction with the environment. We provide distribution-dependent guarantees in the MDL regime, that scale with suboptimality gaps and result in superior dependence on the sample size. We devise an adaptive optimistic algorithm, LCB-DR, that showcases enhanced dependence on the gaps, mirroring the contrast between uniform and optimistic allocation in the multi-armed bandit literature.
  arXiv  Detail & Related papers  (2023-12-20T15:50:16Z)
• Personalized Trajectory Prediction via Distribution Discrimination [78.69458579657189]
  Trarimiy prediction is confronted with the dilemma to capture the multi-modal nature of future dynamics. We present a distribution discrimination (DisDis) method to predict personalized motion patterns. Our method can be integrated with existing multi-modal predictive models as a plug-and-play module.
  arXiv  Detail & Related papers  (2021-07-29T17:42:12Z)
• Robust Generalization despite Distribution Shift via Minimum Discriminating Information [46.164498176119665]
  We introduce a modeling framework where, in addition to training data, we have partial structural knowledge of the shifted test distribution. We employ the principle of minimum discriminating information to embed the available prior knowledge. We obtain explicit generalization bounds with respect to the unknown shifted distribution.
  arXiv  Detail & Related papers  (2021-06-08T15:25:35Z)
• A Brief Introduction to Generative Models [8.031257560764336]
  We introduce and motivate generative modeling as a central task for machine learning. We outline the maximum likelihood approach and how it can be interpreted as minimizing KL-divergence. We explore the alternative adversarial approach which involves studying the differences between an estimating distribution and a real data distribution.
  arXiv  Detail & Related papers  (2021-02-27T16:49:41Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)
• Distributional Reinforcement Learning via Moment Matching [54.16108052278444]
  We formulate a method that learns a finite set of statistics from each return distribution via neural networks. Our method can be interpreted as implicitly matching all orders of moments between a return distribution and its Bellman target. Experiments on the suite of Atari games show that our method outperforms the standard distributional RL baselines.
  arXiv  Detail & Related papers  (2020-07-24T05:18:17Z)
• Implicit Distributional Reinforcement Learning [61.166030238490634]
  implicit distributional actor-critic (IDAC) built on two deep generator networks (DGNs) Semi-implicit actor (SIA) powered by a flexible policy distribution. We observe IDAC outperforms state-of-the-art algorithms on representative OpenAI Gym environments.
  arXiv  Detail & Related papers  (2020-07-13T02:52:18Z)
• A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithms [67.67377846416106]
  We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes. We show that value-based methods such as TD($lambda$) and $Q$-Learning have update rules which are contractive in the space of distributions of functions.
  arXiv  Detail & Related papers  (2020-03-27T05:13:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.