iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learning

• URL: http://arxiv.org/abs/2406.02696v1
• Date: Tue, 4 Jun 2024 18:15:44 GMT
• Title: iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learning
• Authors: Aidan Scannell, Kalle Kujanpää, Yi Zhao, Mohammadreza Nakhaei, Arno Solin, Joni Pajarinen,
• Abstract summary: We propose an efficient representation learning method using only a self-supervised latent-state consistency loss. We achieve high performance and prevent representation collapse by quantizing the latent representation. Our method, named iQRL: implicitly Quantized Reinforcement Learning, is straightforward, compatible with any model-free RL algorithm.
• Score: 24.684363928059113
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning representations for reinforcement learning (RL) has shown much promise for continuous control. We propose an efficient representation learning method using only a self-supervised latent-state consistency loss. Our approach employs an encoder and a dynamics model to map observations to latent states and predict future latent states, respectively. We achieve high performance and prevent representation collapse by quantizing the latent representation such that the rank of the representation is empirically preserved. Our method, named iQRL: implicitly Quantized Reinforcement Learning, is straightforward, compatible with any model-free RL algorithm, and demonstrates excellent performance by outperforming other recently proposed representation learning methods in continuous control benchmarks from DeepMind Control Suite.

Related papers

• Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning [57.67629402360924]
  We introduce the Partially Supervised Reinforcement Learning (PSRL) framework. At the heart of PSRL is the fusion of both supervised and unsupervised learning. We show that PSRL offers a potent balance, enhancing model interpretability while preserving, and often significantly outperforming, the performance benchmarks set by traditional methods.
  arXiv  Detail & Related papers  (2024-02-14T16:23:23Z)
• Light-weight probing of unsupervised representations for Reinforcement Learning [20.638410483549706]
  We study whether linear probing can be a proxy evaluation task for the quality of unsupervised RL representation. We show that the probing tasks are strongly rank correlated with the downstream RL performance on the Atari100k Benchmark. This provides a more efficient method for exploring the space of pretraining algorithms and identifying promising pretraining recipes.
  arXiv  Detail & Related papers  (2022-08-25T21:08:01Z)
• Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning [92.18524491615548]
  Contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL) We study how RL can be empowered by contrastive learning in a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. Under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs.
  arXiv  Detail & Related papers  (2022-07-29T17:29:08Z)
• Learning Bellman Complete Representations for Offline Policy Evaluation [51.96704525783913]
  Two sufficient conditions for sample-efficient OPE are Bellman completeness and coverage. We show our representation enables better OPE compared to previous representation learning methods developed for off-policy RL.
  arXiv  Detail & Related papers  (2022-07-12T21:02:02Z)
• Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning [105.70602423944148]
  We propose a novel method, called value-consistent representation learning (VCR), to learn representations that are directly related to decision-making. Instead of aligning this imagined state with a real state returned by the environment, VCR applies a $Q$-value head on both states and obtains two distributions of action values. It has been demonstrated that our methods achieve new state-of-the-art performance for search-free RL algorithms.
  arXiv  Detail & Related papers  (2022-06-25T03:02:25Z)
• Contrastive Learning as Goal-Conditioned Reinforcement Learning [147.28638631734486]
  In reinforcement learning (RL), it is easier to solve a task if given a good representation. While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable. We show (contrastive) representation learning methods can be cast as RL algorithms in their own right.
  arXiv  Detail & Related papers  (2022-06-15T14:34:15Z)
• Mask-based Latent Reconstruction for Reinforcement Learning [58.43247393611453]
  Mask-based Latent Reconstruction (MLR) is proposed to predict the complete state representations in the latent space from the observations with spatially and temporally masked pixels. Extensive experiments show that our MLR significantly improves the sample efficiency in deep reinforcement learning.
  arXiv  Detail & Related papers  (2022-01-28T13:07:11Z)
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning [3.308743964406687]
  $k$-Step Latent (KSL) is a representation learning method that enforces temporal consistency of representations. KSL produces encoders that generalize better to new tasks unseen during training.
  arXiv  Detail & Related papers  (2021-10-11T00:16:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.