Learning Options via Compression

• URL: http://arxiv.org/abs/2212.04590v1
• Date: Thu, 8 Dec 2022 22:34:59 GMT
• Title: Learning Options via Compression
• Authors: Yiding Jiang, Evan Zheran Liu, Benjamin Eysenbach, Zico Kolter, Chelsea Finn
• Abstract summary: We propose a new objective that combines the maximum likelihood objective with a penalty on the description length of the skills. Our objective learns skills that solve downstream tasks in fewer samples compared to skills learned from only maximizing likelihood.
• Score: 62.55893046218824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Identifying statistical regularities in solutions to some tasks in multi-task reinforcement learning can accelerate the learning of new tasks. Skill learning offers one way of identifying these regularities by decomposing pre-collected experiences into a sequence of skills. A popular approach to skill learning is maximizing the likelihood of the pre-collected experience with latent variable models, where the latent variables represent the skills. However, there are often many solutions that maximize the likelihood equally well, including degenerate solutions. To address this underspecification, we propose a new objective that combines the maximum likelihood objective with a penalty on the description length of the skills. This penalty incentivizes the skills to maximally extract common structures from the experiences. Empirically, our objective learns skills that solve downstream tasks in fewer samples compared to skills learned from only maximizing likelihood. Further, while most prior works in the offline multi-task setting focus on tasks with low-dimensional observations, our objective can scale to challenging tasks with high-dimensional image observations.

Related papers

• Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning [11.260650180067278]
  We introduce a novel approach that combines task-common skills and task-specific skills. A skill assignment matrix is jointly learned. Our findings demonstrate that C-Poly outperforms fully-shared, task-specific, and skill-indistinguishable baselines.
  arXiv  Detail & Related papers  (2023-12-06T02:47:56Z)
• Efficient Rehearsal Free Zero Forgetting Continual Learning using Adaptive Weight Modulation [3.6683171094134805]
  Continual learning involves acquiring knowledge of multiple tasks over an extended period. Most approaches to this problem seek a balance between maximizing performance on the new tasks and minimizing the forgetting of previous tasks. Our approach attempts to maximize the performance of the new task, while ensuring zero forgetting.
  arXiv  Detail & Related papers  (2023-11-26T12:36:05Z)
• Saliency-Regularized Deep Multi-Task Learning [7.3810864598379755]
  Multitask learning enforces multiple learning tasks to share knowledge to improve their generalization abilities. Modern deep multitask learning can jointly learn latent features and task sharing, but they are obscure in task relation. This paper proposes a new multitask learning framework that jointly learns latent features and explicit task relations.
  arXiv  Detail & Related papers  (2022-07-03T20:26:44Z)
• Combining Modular Skills in Multitask Learning [149.8001096811708]
  A modular design encourages neural models to disentangle and recombine different facets of knowledge to generalise more systematically to new tasks. In this work, we assume each task is associated with a subset of latent discrete skills from a (potentially small) inventory. We find that the modular design of a network significantly increases sample efficiency in reinforcement learning and few-shot generalisation in supervised learning.
  arXiv  Detail & Related papers  (2022-02-28T16:07:19Z)
• Variational Multi-Task Learning with Gumbel-Softmax Priors [105.22406384964144]
  Multi-task learning aims to explore task relatedness to improve individual tasks. We propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks.
  arXiv  Detail & Related papers  (2021-11-09T18:49:45Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• Multi-Task Learning with Sequence-Conditioned Transporter Networks [67.57293592529517]
  We aim to solve multi-task learning through the lens of sequence-conditioning and weighted sampling. We propose a new suite of benchmark aimed at compositional tasks, MultiRavens, which allows defining custom task combinations. Second, we propose a vision-based end-to-end system architecture, Sequence-Conditioned Transporter Networks, which augments Goal-Conditioned Transporter Networks with sequence-conditioning and weighted sampling.
  arXiv  Detail & Related papers  (2021-09-15T21:19:11Z)
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks [82.16392072211337]
  We propose a method to discover generalizable skills via automated generation of a diverse set of tasks. As opposed to prior work on unsupervised discovery of skills, our method pairs each skill with a unique task produced by a trainable task generator. A task discriminator defined on the robot behaviors in the generated tasks is jointly trained to estimate the evidence lower bound of the diversity objective. The learned skills can then be composed in a hierarchical reinforcement learning algorithm to solve unseen target tasks.
  arXiv  Detail & Related papers  (2021-06-26T03:41:51Z)
• Learning Transferable Concepts in Deep Reinforcement Learning [0.7161783472741748]
  We show that learning discrete representations of sensory inputs can provide a high-level abstraction that is common across multiple tasks. In particular, we show that it is possible to learn such representations by self-supervision, following an information theoretic approach. Our method is able to learn concepts in locomotive and optimal control tasks that increase the sample efficiency in both known and unknown tasks.
  arXiv  Detail & Related papers  (2020-05-16T04:45:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.