Return-Based Contrastive Representation Learning for Reinforcement Learning

• URL: http://arxiv.org/abs/2102.10960v1
• Date: Mon, 22 Feb 2021 13:04:18 GMT
• Title: Return-Based Contrastive Representation Learning for Reinforcement Learning
• Authors: Guoqing Liu, Chuheng Zhang, Li Zhao, Tao Qin, Jinhua Zhu, Jian Li, Nenghai Yu, Tie-Yan Liu
• Abstract summary: We propose a novel auxiliary task that forces the learnt representations to discriminate state-action pairs with different returns. Our algorithm outperforms strong baselines on complex tasks in Atari games and DeepMind Control suite.
• Score: 126.7440353288838
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, various auxiliary tasks have been proposed to accelerate representation learning and improve sample efficiency in deep reinforcement learning (RL). However, existing auxiliary tasks do not take the characteristics of RL problems into consideration and are unsupervised. By leveraging returns, the most important feedback signals in RL, we propose a novel auxiliary task that forces the learnt representations to discriminate state-action pairs with different returns. Our auxiliary loss is theoretically justified to learn representations that capture the structure of a new form of state-action abstraction, under which state-action pairs with similar return distributions are aggregated together. In low data regime, our algorithm outperforms strong baselines on complex tasks in Atari games and DeepMind Control suite, and achieves even better performance when combined with existing auxiliary tasks.

Related papers

• Offline Multitask Representation Learning for Reinforcement Learning [86.26066704016056]
  We study offline multitask representation learning in reinforcement learning (RL) We propose a new algorithm called MORL for offline multitask representation learning. Our theoretical results demonstrate the benefits of using the learned representation from the upstream offline task instead of directly learning the representation of the low-rank model.
  arXiv  Detail & Related papers  (2024-03-18T08:50:30Z)
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning [57.38874587065694]
  We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks.
  arXiv  Detail & Related papers  (2024-01-17T19:31:21Z)
• Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL [16.792949555151978]
  Learning self-supervised representations using reconstruction or contrastive losses improves performance and sample complexity of image-based and multimodal reinforcement learning (RL) Here, different self-supervised loss functions have distinct advantages and limitations depending on the information density of the underlying sensor modality. We propose Contrastive Reconstructive Aggregated representation Learning (CoRAL), a unified framework enabling us to choose the most appropriate self-supervised loss for each sensor modality.
  arXiv  Detail & Related papers  (2023-02-10T15:57:20Z)
• Composite Learning for Robust and Effective Dense Predictions [81.2055761433725]
  Multi-task learning promises better model generalization on a target task by jointly optimizing it with an auxiliary task. We find that jointly training a dense prediction (target) task with a self-supervised (auxiliary) task can consistently improve the performance of the target task, while eliminating the need for labeling auxiliary tasks.
  arXiv  Detail & Related papers  (2022-10-13T17:59:16Z)
• Reinforcement Learning with Automated Auxiliary Loss Search [34.83123677004838]
  We propose a principled and universal method for learning better representations with auxiliary loss functions. Specifically, we define a general auxiliary loss space of size $7.5 times 1020$ and explore the space with an efficient evolutionary search strategy. Empirical results show that the discovered auxiliary loss significantly improves the performance on both high-dimensional (image) and low-dimensional (vector) unseen tasks.
  arXiv  Detail & Related papers  (2022-10-12T09:24:53Z)
• Provable Benefit of Multitask Representation Learning in Reinforcement Learning [46.11628795660159]
  This paper theoretically characterizes the benefit of representation learning under the low-rank Markov decision process (MDP) model. To the best of our knowledge, this is the first theoretical study that characterizes the benefit of representation learning in exploration-based reward-free multitask reinforcement learning.
  arXiv  Detail & Related papers  (2022-06-13T04:29:02Z)
• Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions [63.773813221460614]
  Generalization across different environments with the same tasks is critical for successful applications of visual reinforcement learning. We propose a novel approach, namely Characteristic Reward Sequence Prediction (CRESP), to extract the task-relevant information. Experiments demonstrate that CRESP significantly improves the performance of generalization on unseen environments.
  arXiv  Detail & Related papers  (2022-05-20T14:52:03Z)
• Exploratory State Representation Learning [63.942632088208505]
  We propose a new approach called XSRL (eXploratory State Representation Learning) to solve the problems of exploration and SRL in parallel. On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations. On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a $k$-step learning progress bonus to form the objective of a discovery policy.
  arXiv  Detail & Related papers  (2021-09-28T10:11:07Z)
• Intrinsically Motivated Self-supervised Learning in Reinforcement Learning [15.809835721792687]
  In vision-based reinforcement learning (RL) tasks, it is prevalent to assign the auxiliary task with a surrogate self-supervised loss. We present a simple yet effective idea to employ self-supervised loss as an intrinsic reward, called Intrinsically Motivated Self-Supervised learning in Reinforcement learning (IM-SSR) We show that the self-supervised loss can be robustness as exploration for novel states and improvement from nuisance elimination.
  arXiv  Detail & Related papers  (2021-06-26T08:43:28Z)
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning [13.36223726517518]
  This work proposes REPresentation And IN Transfer (REPAINT) algorithm for knowledge transfer in deep reinforcement learning. REPAINT not only transfers the representation of a pre-trained teacher policy in the on-policy learning, but also uses an advantage-based experience selection approach to transfer useful samples collected following the teacher policy in the off-policy learning.
  arXiv  Detail & Related papers  (2020-11-24T01:18:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.