Combining Behaviors with the Successor Features Keyboard

• URL: http://arxiv.org/abs/2310.15940v1
• Date: Tue, 24 Oct 2023 15:35:54 GMT
• Title: Combining Behaviors with the Successor Features Keyboard
• Authors: Wilka Carvalho, Andre Saraiva, Angelos Filos, Andrew Kyle Lampinen, Loic Matthey, Richard L. Lewis, Honglak Lee, Satinder Singh, Danilo J. Rezende, Daniel Zoran
• Abstract summary: "Successor Features Keyboard" (SFK) enables transfer with discovered state-features and task encodings. We achieve the first demonstration of transfer with SFs in a challenging 3D environment.
• Score: 55.983751286962985
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Option Keyboard (OK) was recently proposed as a method for transferring behavioral knowledge across tasks. OK transfers knowledge by adaptively combining subsets of known behaviors using Successor Features (SFs) and Generalized Policy Improvement (GPI). However, it relies on hand-designed state-features and task encodings which are cumbersome to design for every new environment. In this work, we propose the "Successor Features Keyboard" (SFK), which enables transfer with discovered state-features and task encodings. To enable discovery, we propose the "Categorical Successor Feature Approximator" (CSFA), a novel learning algorithm for estimating SFs while jointly discovering state-features and task encodings. With SFK and CSFA, we achieve the first demonstration of transfer with SFs in a challenging 3D environment where all the necessary representations are discovered. We first compare CSFA against other methods for approximating SFs and show that only CSFA discovers representations compatible with SF&GPI at this scale. We then compare SFK against transfer learning baselines and show that it transfers most quickly to long-horizon tasks.

Related papers

• Trustworthy Image Semantic Communication with GenAI: Explainablity, Controllability, and Efficiency [59.15544887307901]
  Image semantic communication (ISC) has garnered significant attention for its potential to achieve high efficiency in visual content transmission. Existing ISC systems based on joint source-channel coding face challenges in interpretability, operability, and compatibility. We propose a novel trustworthy ISC framework that employs Generative Artificial Intelligence (GenAI) for multiple downstream inference tasks.
  arXiv  Detail & Related papers  (2024-08-07T14:32:36Z)
• SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning [89.04776523010409]
  This paper studies the transfer reinforcement learning (RL) problem where multiple RL problems have different reward functions but share the same underlying transition dynamics. In this setting, the Q-function of each RL problem (task) can be decomposed into a successor feature (SF) and a reward mapping. We establish the first convergence analysis with provable generalization guarantees for SF-DQN with GPI.
  arXiv  Detail & Related papers  (2024-05-24T20:30:14Z)
• Facing Unknown: Open-World Encrypted Traffic Classification Based on Contrastive Pre-Training [5.318006462723139]
  We propose a novel Open-World Contrastive Pre-training (OWCP) framework for this. OWCP performs contrastive pre-training to obtain a robust feature representation. We conduct comprehensive ablation studies and sensitivity analyses to validate each integral component of OWCP.
  arXiv  Detail & Related papers  (2023-08-31T17:04:20Z)
• Composing Task Knowledge with Modular Successor Feature Approximators [60.431769158952626]
  We present a novel neural network architecture, "Modular Successor Feature Approximators" (MSFA) MSFA is able to better generalize compared to baseline architectures for learning SFs and modular architectures.
  arXiv  Detail & Related papers  (2023-01-28T23:04:07Z)
• Task Relabelling for Multi-task Transfer using Successor Features [0.0]
  Successor Features (SFs) proposes a mechanism that allows learning policies that are not tied to any particular reward function. In this work we investigate how SFs may be pre-trained without observing any reward in a custom environment that features resource collection, traps and crafting.
  arXiv  Detail & Related papers  (2022-05-20T13:29:29Z)
• CATCH: Context-based Meta Reinforcement Learning for Transferrable Architecture Search [102.67142711824748]
  CATCH is a novel Context-bAsed meTa reinforcement learning algorithm for transferrable arChitecture searcH. The combination of meta-learning and RL allows CATCH to efficiently adapt to new tasks while being agnostic to search spaces. It is also capable of handling cross-domain architecture search as competitive networks on ImageNet, COCO, and Cityscapes are identified.
  arXiv  Detail & Related papers  (2020-07-18T09:35:53Z)
• UR2KiD: Unifying Retrieval, Keypoint Detection, and Keypoint Description without Local Correspondence Supervision [16.68130648568593]
  Three related tasks, namely keypoint detection, description, and image retrieval can be jointly tackled using a single unified framework. By leveraging diverse information from sequential layers of a standard ResNet-based architecture, we are able to extract keypoints and descriptors that encode local information. Global information for image retrieval is encoded in an end-to-end pipeline, based on pooling of the aforementioned local responses.
  arXiv  Detail & Related papers  (2020-01-20T21:01:38Z)
• Universal Successor Features for Transfer Reinforcement Learning [77.27304854836645]
  We propose Universal Successor Features (USFs) to capture the underlying dynamics of the environment. We show that USFs is compatible with any RL algorithm that learns state values using a temporal difference method.
  arXiv  Detail & Related papers  (2020-01-05T03:41:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.