Zero-Shot Reinforcement Learning via Function Encoders

• URL: http://arxiv.org/abs/2401.17173v3
• Date: Fri, 21 Mar 2025 14:37:37 GMT
• Title: Zero-Shot Reinforcement Learning via Function Encoders
• Authors: Tyler Ingebrand, Amy Zhang, Ufuk Topcu,
• Abstract summary: We introduce the function encoder, a representation learning algorithm which represents a function as a weighted combination of learned, non-linear basis functions.<n>By using a function encoder to represent the reward function or the transition function, the agent has information on how current task relates to previously seen tasks.<n>We demonstrate state-of-the-art data efficiency, stability, and training stability in three RL fields by augmenting basic RL algorithms with a function task representation.
• Score: 23.57570432980556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although reinforcement learning (RL) can solve many challenging sequential decision making problems, achieving zero-shot transfer across related tasks remains a challenge. The difficulty lies in finding a good representation for the current task so that the agent understands how it relates to previously seen tasks. To achieve zero-shot transfer, we introduce the function encoder, a representation learning algorithm which represents a function as a weighted combination of learned, non-linear basis functions. By using a function encoder to represent the reward function or the transition function, the agent has information on how the current task relates to previously seen tasks via a coherent vector representation. Thus, the agent is able to achieve transfer between related tasks at run time with no additional training. We demonstrate state-of-the-art data efficiency, asymptotic performance, and training stability in three RL fields by augmenting basic RL algorithms with a function encoder task representation.

Related papers

• Lifelong Reinforcement Learning with Similarity-Driven Weighting by Large Models [4.265969066588072]
  We propose a novel framework, SDW, which leverages large-language-model-generated dynamic functions to control the training process. The core of SDW lies in two functions pre-generated by large models: the task similarity function and the weight computation function. Experimental results on Atari and MiniHack sequential tasks demonstrate that SDW significantly outperforms existing lifelong reinforcement learning methods.
  arXiv  Detail & Related papers  (2025-03-17T08:36:16Z)
• Towards Understanding the Benefit of Multitask Representation Learning in Decision Process [47.57497888077687]
  Multitask Representation Learning (MRL) has emerged as a prevalent technique to improve sample efficiency in Reinforcement Learning (RL) Our work tries to fill the gap by extending the analysis to textitunknown non-linear representations, giving a comprehensive analysis for its mechanism in online and transfer learning setting. We formally prove that this approach yields a regret upper bound that outperforms the lower bound associated with learning $M$ separate tasks, marking the first demonstration of MRL's efficacy in a general function class.
  arXiv  Detail & Related papers  (2025-03-01T04:29:22Z)
• Learning Task Representations from In-Context Learning [73.72066284711462]
  Large language models (LLMs) have demonstrated remarkable proficiency in in-context learning. We introduce an automated formulation for encoding task information in ICL prompts as a function of attention heads. We show that our method's effectiveness stems from aligning the distribution of the last hidden state with that of an optimally performing in-context-learned model.
  arXiv  Detail & Related papers  (2025-02-08T00:16:44Z)
• Reinforcement Learning with Action Sequence for Data-Efficient Robot Learning [62.3886343725955]
  We introduce a novel RL algorithm that learns a critic network that outputs Q-values over a sequence of actions. By explicitly training the value functions to learn the consequence of executing a series of current and future actions, our algorithm allows for learning useful value functions from noisy trajectories.
  arXiv  Detail & Related papers  (2024-11-19T01:23:52Z)
• How Feature Learning Can Improve Neural Scaling Laws [86.9540615081759]
  We develop a solvable model of neural scaling laws beyond the kernel limit. We show how performance scales with model size, training time, and the total amount of available data.
  arXiv  Detail & Related papers  (2024-09-26T14:05:32Z)
• Evaluating the structure of cognitive tasks with transfer learning [67.22168759751541]
  This study investigates the transferability of deep learning representations between different EEG decoding tasks. We conduct extensive experiments using state-of-the-art decoding models on two recently released EEG datasets.
  arXiv  Detail & Related papers  (2023-07-28T14:51:09Z)
• Functional Knowledge Transfer with Self-supervised Representation Learning [11.566644244783305]
  This work investigates the unexplored usability of self-supervised representation learning in the direction of functional knowledge transfer. In this work, functional knowledge transfer is achieved by joint optimization of self-supervised learning pseudo task and supervised learning task.
  arXiv  Detail & Related papers  (2023-03-12T21:14:59Z)
• Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributions [63.773813221460614]
  Generalization across different environments with the same tasks is critical for successful applications of visual reinforcement learning. We propose a novel approach, namely Characteristic Reward Sequence Prediction (CRESP), to extract the task-relevant information. Experiments demonstrate that CRESP significantly improves the performance of generalization on unseen environments.
  arXiv  Detail & Related papers  (2022-05-20T14:52:03Z)
• Adaptive Transfer Learning on Graph Neural Networks [4.233435459239147]
  Graph neural networks (GNNs) are widely used to learn a powerful representation of graph-structured data. Recent work demonstrates that transferring knowledge from self-supervised tasks to downstream tasks could further improve graph representation. We propose a new transfer learning paradigm on GNNs which could effectively leverage self-supervised tasks as auxiliary tasks to help the target task.
  arXiv  Detail & Related papers  (2021-07-19T11:46:28Z)
• Multi-task Supervised Learning via Cross-learning [102.64082402388192]
  We consider a problem known as multi-task learning, consisting of fitting a set of regression functions intended for solving different tasks. In our novel formulation, we couple the parameters of these functions, so that they learn in their task specific domains while staying close to each other. This facilitates cross-fertilization in which data collected across different domains help improving the learning performance at each other task.
  arXiv  Detail & Related papers  (2020-10-24T21:35:57Z)
• Function Contrastive Learning of Transferable Meta-Representations [38.31692245188669]
  We study the implications of joint training on the transferability of the meta-representations. We propose a decoupled encoder-decoder approach to supervised meta-learning.
  arXiv  Detail & Related papers  (2020-10-14T13:50:22Z)
• Exploring and Predicting Transferability across NLP Tasks [115.6278033699853]
  We study the transferability between 33 NLP tasks across three broad classes of problems. Our results show that transfer learning is more beneficial than previously thought. We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task.
  arXiv  Detail & Related papers  (2020-05-02T09:39:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.