R3L: Relative Representations for Reinforcement Learning

• URL: http://arxiv.org/abs/2404.12917v3
• Date: Tue, 18 Feb 2025 15:17:38 GMT
• Title: R3L: Relative Representations for Reinforcement Learning
• Authors: Antonio Pio Ricciardi, Valentino Maiorca, Luca Moschella, Riccardo Marin, Emanuele Rodolà,
• Abstract summary: It is known that variations in input domains (e.g., different panorama colors due to seasonal changes) can disrupt agent performance.<n>Recent advancements in the field of representation learning have demonstrated the possibility of combining components to create new models.<n>We adapt this framework to the Visual Reinforcement Learning setting, allowing to combine agents components to create new agents capable of effectively handling novel visual-task pairs.
• Score: 17.76990521486307
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Reinforcement Learning is a popular and powerful framework that takes full advantage of the Deep Learning breakthrough. It is known that variations in input domains (e.g., different panorama colors due to seasonal changes) or task domains (e.g., altering the target speed of a car) can disrupt agent performance, necessitating new training for each variation. Recent advancements in the field of representation learning have demonstrated the possibility of combining components from different neural networks to create new models in a zero-shot fashion. In this paper, we build upon relative representations, a framework that maps encoder embeddings to a universal space. We adapt this framework to the Visual Reinforcement Learning setting, allowing to combine agents components to create new agents capable of effectively handling novel visual-task pairs not encountered during training. Our findings highlight the potential for model reuse, significantly reducing the need for retraining and, consequently, the time and computational resources required.

Related papers

• Mapping representations in Reinforcement Learning via Semantic Alignment for Zero-Shot Stitching [17.76990521486307]
  Deep Reinforcement Learning models often fail to generalize when even small changes occur in the environment's observations or task requirements. We propose a zero-shot method for mapping between latent spaces across different agents trained on different visual and task variations. We empirically demonstrate zero-shot stitching performance on the CarRacing environment with changing background and task.
  arXiv  Detail & Related papers  (2025-02-26T22:06:00Z)
• ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI [44.77897322913095]
  We present ReLIC, a new approach for in-context reinforcement learning for embodied agents. With ReLIC, agents are capable of adapting to new environments using 64,000 steps of in-context experience. We find that ReLIC is capable of few-shot imitation learning despite never being trained with expert demonstrations.
  arXiv  Detail & Related papers  (2024-10-03T17:58:11Z)
• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• Reusable Architecture Growth for Continual Stereo Matching [92.36221737921274]
  We introduce a Reusable Architecture Growth (RAG) framework to learn new scenes continually in both supervised and self-supervised manners. RAG can maintain high reusability during growth by reusing previous units while obtaining good performance. We also present a Scene Router module to adaptively select the scene-specific architecture path at inference.
  arXiv  Detail & Related papers  (2024-03-30T13:24:58Z)
• Vision-Language Models Provide Promptable Representations for Reinforcement Learning [67.40524195671479]
  We propose a novel approach that uses the vast amounts of general and indexable world knowledge encoded in vision-language models (VLMs) pre-trained on Internet-scale data for embodied reinforcement learning (RL) We show that our approach can use chain-of-thought prompting to produce representations of common-sense semantic reasoning, improving policy performance in novel scenes by 1.5 times.
  arXiv  Detail & Related papers  (2024-02-05T00:48:56Z)
• Look-Ahead Selective Plasticity for Continual Learning of Visual Tasks [9.82510084910641]
  We propose a new mechanism that takes place during task boundaries, i.e., when one task finishes and another starts. We evaluate the proposed methods on benchmark computer vision datasets including CIFAR10 and TinyImagenet.
  arXiv  Detail & Related papers  (2023-11-02T22:00:23Z)
• Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learning [7.972204774778987]
  In real-world robotics applications, Reinforcement Learning (RL) agents are often unable to generalise to environment variations that were not observed during training. We introduce TEmporal Disentanglement (TED), a self-supervised auxiliary task that leads to disentangled representations using the sequential nature of RL observations. We find empirically that RL algorithms with TED as an auxiliary task adapt more quickly to changes in environment variables with continued training compared to state-of-the-art representation learning methods.
  arXiv  Detail & Related papers  (2022-07-12T11:46:49Z)
• RLFlow: Optimising Neural Network Subgraph Transformation with World Models [0.0]
  We propose a model-based agent which learns to optimise the architecture of neural networks by performing a sequence of subgraph transformations to reduce model runtime. We show our approach can match the performance of state of the art on common convolutional networks and outperform those by up to 5% on transformer-style architectures.
  arXiv  Detail & Related papers  (2022-05-03T11:52:54Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)
• Curious Representation Learning for Embodied Intelligence [81.21764276106924]
  Self-supervised representation learning has achieved remarkable success in recent years. Yet to build truly intelligent agents, we must construct representation learning algorithms that can learn from environments. We propose a framework, curious representation learning, which jointly learns a reinforcement learning policy and a visual representation model.
  arXiv  Detail & Related papers  (2021-05-03T17:59:20Z)
• Meta-Learning with Variational Bayes [0.0]
  We introduce a new approach to address the more general problem of generative meta-learning. Our contribution leverages the AEVB framework and mean-field variational Bayes, and creates fast-adapting latent-space generative models. At the heart of our contribution is a new result, showing that for a broad class of deep generative latent variable models, the relevant VB updates do not depend on any generative neural network.
  arXiv  Detail & Related papers  (2021-03-03T09:02:01Z)
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning [102.36450942613091]
  We propose an inverse reinforcement learning algorithm, called emphinverse temporal difference learning (ITD) We show how to seamlessly integrate ITD with learning from online environment interactions, arriving at a novel algorithm for reinforcement learning with demonstrations, called $Psi Phi$-learning.
  arXiv  Detail & Related papers  (2021-02-24T21:12:09Z)
• Essentials for Class Incremental Learning [43.306374557919646]
  Class-incremental learning results on CIFAR-100 and ImageNet improve over the state-of-the-art by a large margin, while keeping the approach simple.
  arXiv  Detail & Related papers  (2021-02-18T18:01:06Z)
• Graph-Based Neural Network Models with Multiple Self-Supervised Auxiliary Tasks [79.28094304325116]
  Graph Convolutional Networks are among the most promising approaches for capturing relationships among structured data points. We propose three novel self-supervised auxiliary tasks to train graph-based neural network models in a multi-task fashion.
  arXiv  Detail & Related papers  (2020-11-14T11:09:51Z)
• Reward Propagation Using Graph Convolutional Networks [61.32891095232801]
  We propose a new framework for learning potential functions by leveraging ideas from graph representation learning. Our approach relies on Graph Convolutional Networks which we use as a key ingredient in combination with the probabilistic inference view of reinforcement learning.
  arXiv  Detail & Related papers  (2020-10-06T04:38:16Z)
• Bridging the Imitation Gap by Adaptive Insubordination [88.35564081175642]
  We show that when the teaching agent makes decisions with access to privileged information, this information is marginalized during imitation learning. We propose 'Adaptive Insubordination' (ADVISOR) to address this gap. ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration.
  arXiv  Detail & Related papers  (2020-07-23T17:59:57Z)
• Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory [79.42778415729475]
  We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning. We show how to extend the architecture of a simple RNN by separating its hidden state into different modules. We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
  arXiv  Detail & Related papers  (2020-06-29T08:35:49Z)
• ELSIM: End-to-end learning of reusable skills through intrinsic motivation [0.0]
  We present a novel reinforcement learning architecture which hierarchically learns and represents self-generated skills in an end-to-end way. With this architecture, an agent focuses only on task-rewarded skills while keeping the learning process of skills bottom-up.
  arXiv  Detail & Related papers  (2020-06-23T11:20:46Z)
• Towards Backward-Compatible Representation Learning [86.39292571306395]
  We propose a way to learn visual features that are compatible with previously computed ones even when they have different dimensions. This enables visual search systems to bypass computing new features for all previously seen images when updating the embedding models. We propose a framework to train embedding models, called backward-compatible training (BCT), as a first step towards backward compatible representation learning.
  arXiv  Detail & Related papers  (2020-03-26T14:34:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.