SLIM: Skill Learning with Multiple Critics

• URL: http://arxiv.org/abs/2402.00823v2
• Date: Thu, 21 Mar 2024 10:21:37 GMT
• Title: SLIM: Skill Learning with Multiple Critics
• Authors: David Emukpere, Bingbing Wu, Julien Perez, Jean-Michel Renders,
• Abstract summary: Self-supervised skill learning aims to acquire useful behaviors that leverage the underlying dynamics of the environment. Latent variable models, based on mutual information, have been successful in this task but still struggle in the context of robotic manipulation. We introduce SLIM, a multi-critic learning approach for skill discovery with a particular focus on robotic manipulation.
• Score: 8.645929825516818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised skill learning aims to acquire useful behaviors that leverage the underlying dynamics of the environment. Latent variable models, based on mutual information maximization, have been successful in this task but still struggle in the context of robotic manipulation. As it requires impacting a possibly large set of degrees of freedom composing the environment, mutual information maximization fails alone in producing useful and safe manipulation behaviors. Furthermore, tackling this by augmenting skill discovery rewards with additional rewards through a naive combination might fail to produce desired behaviors. To address this limitation, we introduce SLIM, a multi-critic learning approach for skill discovery with a particular focus on robotic manipulation. Our main insight is that utilizing multiple critics in an actor-critic framework to gracefully combine multiple reward functions leads to a significant improvement in latent-variable skill discovery for robotic manipulation while overcoming possible interference occurring among rewards which hinders convergence to useful skills. Furthermore, in the context of tabletop manipulation, we demonstrate the applicability of our novel skill discovery approach to acquire safe and efficient motor primitives in a hierarchical reinforcement learning fashion and leverage them through planning, significantly surpassing baseline approaches for skill discovery.

Related papers

• SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions [48.003320766433966]
  This work introduces Skill Discovery from Local Dependencies (Skild) Skild develops a novel skill learning objective that explicitly encourages the mastering of skills that induce different interactions within an environment. We evaluate Skild in several domains with challenging, long-horizon sparse reward tasks including a realistic simulated household robot domain.
  arXiv  Detail & Related papers  (2024-10-24T04:01:59Z)
• Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning [39.991887534269445]
  Disentangled Unsupervised Skill Discovery (DUSDi) is a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. DUSDi decomposes skills into disentangled components, where each skill component only affects one factor of the state space. DUSDi successfully learns disentangled skills, and significantly outperforms previous skill discovery methods when it comes to applying the learned skills to solve downstream tasks.
  arXiv  Detail & Related papers  (2024-10-15T04:13:20Z)
• Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generation [17.222197596599685]
  We propose a Skill Learning approach that discovers composable behaviors by solving a large number of autonomously generated tasks. Our method learns skills allowing the robot to consistently and robustly interact with objects in its environment. The learned skills can be used to solve a set of unseen manipulation tasks, in simulation as well as on a real robotic platform.
  arXiv  Detail & Related papers  (2024-10-07T09:19:13Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Behavior Contrastive Learning for Unsupervised Skill Discovery [75.6190748711826]
  We propose a novel unsupervised skill discovery method through contrastive learning among behaviors. Under mild assumptions, our objective maximizes the MI between different behaviors based on the same skill. Our method implicitly increases the state entropy to obtain better state coverage.
  arXiv  Detail & Related papers  (2023-05-08T06:02:11Z)
• Learning Options via Compression [62.55893046218824]
  We propose a new objective that combines the maximum likelihood objective with a penalty on the description length of the skills. Our objective learns skills that solve downstream tasks in fewer samples compared to skills learned from only maximizing likelihood.
  arXiv  Detail & Related papers  (2022-12-08T22:34:59Z)
• Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions [19.626042478612572]
  We propose a cooperative adversarial method for obtaining versatile policies with controllable skill sets from unlabeled datasets. We show that by utilizing unsupervised skill discovery in the generative imitation learning framework, novel and useful skills emerge with successful task fulfillment. Finally, the obtained versatile policies are tested on an agile quadruped robot called Solo 8 and present faithful replications of diverse skills encoded in the demonstrations.
  arXiv  Detail & Related papers  (2022-09-16T12:49:04Z)
• Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery [22.32327908453603]
  Current reinforcement learning (RL) in robotics often experiences difficulty in generalizing to new downstream tasks. We propose a framework that pre-trains the agent in a task-agnostic manner without access to the task-specific reward. We show that our approach achieves the most diverse interacting behavior and significantly improves sample efficiency in downstream tasks.
  arXiv  Detail & Related papers  (2022-04-29T06:57:46Z)
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks [82.16392072211337]
  We propose a method to discover generalizable skills via automated generation of a diverse set of tasks. As opposed to prior work on unsupervised discovery of skills, our method pairs each skill with a unique task produced by a trainable task generator. A task discriminator defined on the robot behaviors in the generated tasks is jointly trained to estimate the evidence lower bound of the diversity objective. The learned skills can then be composed in a hierarchical reinforcement learning algorithm to solve unseen target tasks.
  arXiv  Detail & Related papers  (2021-06-26T03:41:51Z)
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning [81.12201426668894]
  We develop efficient reinforcement learning methods that acquire diverse skills without any reward function, and then repurpose these skills for downstream tasks. We show that our proposed algorithm provides substantial improvement in learning efficiency, making reward-free real-world training feasible. We also demonstrate that the learned skills can be composed using model predictive control for goal-oriented navigation, without any additional training.
  arXiv  Detail & Related papers  (2020-04-27T17:38:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.