Pretrained Visual Representations in Reinforcement Learning

• URL: http://arxiv.org/abs/2407.17238v1
• Date: Wed, 24 Jul 2024 12:53:26 GMT
• Title: Pretrained Visual Representations in Reinforcement Learning
• Authors: Emlyn Williams, Athanasios Polydoros,
• Abstract summary: This paper compares the performance of visual reinforcement learning algorithms that train a convolutional neural network (CNN) from scratch with those that utilize pre-trained visual representations (PVRs) We evaluate the Dormant Ratio Minimization (DRM) algorithm, a state-of-the-art visual RL method, against three PVRs: ResNet18, DINOv2, and Visual Cortex (VC)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual reinforcement learning (RL) has made significant progress in recent years, but the choice of visual feature extractor remains a crucial design decision. This paper compares the performance of RL algorithms that train a convolutional neural network (CNN) from scratch with those that utilize pre-trained visual representations (PVRs). We evaluate the Dormant Ratio Minimization (DRM) algorithm, a state-of-the-art visual RL method, against three PVRs: ResNet18, DINOv2, and Visual Cortex (VC). We use the Metaworld Push-v2 and Drawer-Open-v2 tasks for our comparison. Our results show that the choice of training from scratch compared to using PVRs for maximising performance is task-dependent, but PVRs offer advantages in terms of reduced replay buffer size and faster training times. We also identify a strong correlation between the dormant ratio and model performance, highlighting the importance of exploration in visual RL. Our study provides insights into the trade-offs between training from scratch and using PVRs, informing the design of future visual RL algorithms.

Related papers

• The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning [8.36595587335589]
  Visual Reinforcement Learning methods often require extensive amounts of data. Model-based RL (MBRL) offers a potential solution with efficient data utilization through planning. MBRL lacks generalization capabilities for real-world tasks.
  arXiv  Detail & Related papers  (2024-11-15T13:21:26Z)
• Pretraining the Vision Transformer using self-supervised methods for vision based Deep Reinforcement Learning [0.0]
  We study pretraining a Vision Transformer using several state-of-the-art self-supervised methods and assess the quality of the learned representations. Our results show that all methods are effective in learning useful representations and avoiding representational collapse. The encoder pretrained with the temporal order verification task shows the best results across all experiments.
  arXiv  Detail & Related papers  (2022-09-22T10:18:59Z)
• Light-weight probing of unsupervised representations for Reinforcement Learning [20.638410483549706]
  We study whether linear probing can be a proxy evaluation task for the quality of unsupervised RL representation. We show that the probing tasks are strongly rank correlated with the downstream RL performance on the Atari100k Benchmark. This provides a more efficient method for exploring the space of pretraining algorithms and identifying promising pretraining recipes.
  arXiv  Detail & Related papers  (2022-08-25T21:08:01Z)
• Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning [105.70602423944148]
  We propose a novel method, called value-consistent representation learning (VCR), to learn representations that are directly related to decision-making. Instead of aligning this imagined state with a real state returned by the environment, VCR applies a $Q$-value head on both states and obtains two distributions of action values. It has been demonstrated that our methods achieve new state-of-the-art performance for search-free RL algorithms.
  arXiv  Detail & Related papers  (2022-06-25T03:02:25Z)
• SCVRL: Shuffled Contrastive Video Representation Learning [28.06521069427918]
  SCVRL is a contrastive-based framework for self-supervised learning for videos. We reformulate the popular shuffling pretext task within a modern contrastive learning paradigm. Our network has a natural capacity to learn motion in self-supervised settings and achieves strong performance, outperforming CVRL on four benchmarks.
  arXiv  Detail & Related papers  (2022-05-24T01:24:47Z)
• Offline Visual Representation Learning for Embodied Navigation [50.442660137987275]
  offline pretraining of visual representations with self-supervised learning (SSL) Online finetuning of visuomotor representations on specific tasks with image augmentations under long learning schedules.
  arXiv  Detail & Related papers  (2022-04-27T23:22:43Z)
• Visual Alignment Constraint for Continuous Sign Language Recognition [74.26707067455837]
  Vision-based Continuous Sign Language Recognition aims to recognize unsegmented gestures from image sequences. In this work, we revisit the overfitting problem in recent CTC-based CSLR works and attribute it to the insufficient training of the feature extractor. We propose a Visual Alignment Constraint (VAC) to enhance the feature extractor with more alignment supervision.
  arXiv  Detail & Related papers  (2021-04-06T07:24:58Z)
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings [89.63764845984076]
  We present Stored Embeddings for Efficient Reinforcement Learning (SEER) SEER is a simple modification of existing off-policy deep reinforcement learning methods. We show that SEER does not degrade the performance of RLizable agents while significantly saving computation and memory.
  arXiv  Detail & Related papers  (2021-03-04T08:14:10Z)
• Decoupling Representation Learning from Reinforcement Learning [89.82834016009461]
  We introduce an unsupervised learning task called Augmented Temporal Contrast (ATC) ATC trains a convolutional encoder to associate pairs of observations separated by a short time difference. In online RL experiments, we show that training the encoder exclusively using ATC matches or outperforms end-to-end RL.
  arXiv  Detail & Related papers  (2020-09-14T19:11:13Z)
• Learning to Prune Deep Neural Networks via Reinforcement Learning [64.85939668308966]
  PuRL is a deep reinforcement learning based algorithm for pruning neural networks. It achieves sparsity and accuracy comparable to current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-09T13:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.