Balancing Both Behavioral Quality and Diversity in Unsupervised Skill Discovery

• URL: http://arxiv.org/abs/2309.17203v2
• Date: Sun, 19 May 2024 10:11:54 GMT
• Title: Balancing Both Behavioral Quality and Diversity in Unsupervised Skill Discovery
• Authors: Xin Liu, Yaran Chen, Dongbin Zhao,
• Abstract summary: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. We propose textbfContrastive textbfmulti-objective textbfSkill textbfDiscovery textbf(ComSD) which discovers exploratory and diverse behaviors through a novel intrinsic incentive, named contrastive multi-objective reward.
• Score: 12.277005054008017
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. Unsupervised skill discovery seeks to dig out diverse and exploratory skills without extrinsic reward, with the discovered skills efficiently adapting to multiple downstream tasks in various ways. However, recent advanced methods struggle to well balance behavioral exploration and diversity, particularly when the agent dynamics are complex and potential skills are hard to discern (e.g., robot behavior discovery). In this paper, we propose \textbf{Co}ntrastive \textbf{m}ulti-objective \textbf{S}kill \textbf{D}iscovery \textbf{(ComSD)} which discovers exploratory and diverse behaviors through a novel intrinsic incentive, named contrastive multi-objective reward. It contains a novel diversity reward based on contrastive learning to effectively drive agents to discern existing skills, and a particle-based exploration reward to access and learn new behaviors. Moreover, a novel dynamic weighting mechanism between the above two rewards is proposed for diversity-exploration balance, which further improves behavioral quality. Extensive experiments and analysis demonstrate that ComSD can generate diverse behaviors at different exploratory levels for complex multi-joint robots, enabling state-of-the-art performance across 32 challenging downstream adaptation tasks, which recent advanced methods cannot. Codes will be opened after publication.

Related papers

• Human-Aligned Skill Discovery: Balancing Behaviour Exploration and Alignment [14.948610521764415]
  We propose Human-aligned Skill Discovery (HaSD) to discover safer, more aligned skills. HaSD simultaneously optimises skill diversity and alignment with human values. We demonstrate its effectiveness in both 2D navigation and SafetyGymnasium environments.
  arXiv  Detail & Related papers  (2025-01-29T06:14:27Z)
• SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions [48.003320766433966]
  This work introduces Skill Discovery from Local Dependencies (Skild) Skild develops a novel skill learning objective that explicitly encourages the mastering of skills that induce different interactions within an environment. We evaluate Skild in several domains with challenging, long-horizon sparse reward tasks including a realistic simulated household robot domain.
  arXiv  Detail & Related papers  (2024-10-24T04:01:59Z)
• Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning [39.991887534269445]
  Disentangled Unsupervised Skill Discovery (DUSDi) is a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. DUSDi decomposes skills into disentangled components, where each skill component only affects one factor of the state space. DUSDi successfully learns disentangled skills, and significantly outperforms previous skill discovery methods when it comes to applying the learned skills to solve downstream tasks.
  arXiv  Detail & Related papers  (2024-10-15T04:13:20Z)
• Language Guided Skill Discovery [56.84356022198222]
  We introduce Language Guided Skill Discovery (LGSD) to maximize semantic diversity between skills. LGSD takes user prompts as input and outputs a set of semantically distinctive skills. We demonstrate that LGSD enables legged robots to visit different user-intended areas on a plane by simply changing the prompt.
  arXiv  Detail & Related papers  (2024-06-07T04:25:38Z)
• DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning [84.22561239481901]
  We propose a new approach that enables agents to learn whether their behaviors should be consistent with that of other agents. We evaluate DCIR in multiple environments including Multi-agent Particle, Google Research Football and StarCraft II Micromanagement.
  arXiv  Detail & Related papers  (2023-12-10T06:03:57Z)
• APART: Diverse Skill Discovery using All Pairs with Ascending Reward and DropouT [16.75358022780262]
  We study diverse skill discovery in reward-free environments, aiming to discover all possible skills in simple grid-world environments. This problem is formulated as mutual training of skills using an intrinsic reward and a discriminator trained to predict a skill given its trajectory. Our initial solution replaces the standard one-vs-all (softmax) discriminator with a one-vs-one (all pairs) discriminator and combines it with a novel intrinsic reward function and a dropout regularization technique.
  arXiv  Detail & Related papers  (2023-08-24T08:46:43Z)
• Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding [52.723297744257536]
  Pre-trained language models (LMs) have shown effectiveness in scientific literature understanding tasks. We propose a multi-task contrastive learning framework, SciMult, to facilitate common knowledge sharing across different literature understanding tasks.
  arXiv  Detail & Related papers  (2023-05-23T16:47:22Z)
• Behavior Contrastive Learning for Unsupervised Skill Discovery [75.6190748711826]
  We propose a novel unsupervised skill discovery method through contrastive learning among behaviors. Under mild assumptions, our objective maximizes the MI between different behaviors based on the same skill. Our method implicitly increases the state entropy to obtain better state coverage.
  arXiv  Detail & Related papers  (2023-05-08T06:02:11Z)
• System Neural Diversity: Measuring Behavioral Heterogeneity in Multi-Agent Learning [8.280943341629161]
  We introduce System Neural Diversity (SND): a measure of behavioral heterogeneity in multi-agent systems. We show how SND allows us to measure latent resilience skills acquired by the agents, while other proxies, such as task performance (reward), fail. We demonstrate how this paradigm can be used to bootstrap the exploration phase, finding optimal policies faster.
  arXiv  Detail & Related papers  (2023-05-03T13:58:13Z)
• Controlled Diversity with Preference : Towards Learning a Diverse Set of Desired Skills [15.187171070594935]
  We propose Controlled Diversity with Preference (CDP), a collaborative human-guided mechanism for an agent to learn a set of skills that is diverse as well as desirable. The key principle is to restrict the discovery of skills to those regions that are deemed to be desirable as per a preference model trained using human preference labels on trajectory pairs. We evaluate our approach on 2D navigation and Mujoco environments and demonstrate the ability to discover diverse, yet desirable skills.
  arXiv  Detail & Related papers  (2023-03-07T03:37:47Z)
• Controllability-Aware Unsupervised Skill Discovery [94.19932297743439]
  We introduce a novel unsupervised skill discovery method, Controllability-aware Skill Discovery (CSD), which actively seeks complex, hard-to-control skills without supervision. The key component of CSD is a controllability-aware distance function, which assigns larger values to state transitions that are harder to achieve with the current skills. Our experimental results in six robotic manipulation and locomotion environments demonstrate that CSD can discover diverse complex skills with no supervision.
  arXiv  Detail & Related papers  (2023-02-10T08:03:09Z)
• Learning Options via Compression [62.55893046218824]
  We propose a new objective that combines the maximum likelihood objective with a penalty on the description length of the skills. Our objective learns skills that solve downstream tasks in fewer samples compared to skills learned from only maximizing likelihood.
  arXiv  Detail & Related papers  (2022-12-08T22:34:59Z)
• LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning [122.47938710284784]
  We propose a novel framework for learning dynamic subtask assignment (LDSA) in cooperative MARL. To reasonably assign agents to different subtasks, we propose an ability-based subtask selection strategy. We show that LDSA learns reasonable and effective subtask assignment for better collaboration.
  arXiv  Detail & Related papers  (2022-05-05T10:46:16Z)
• Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery [22.32327908453603]
  Current reinforcement learning (RL) in robotics often experiences difficulty in generalizing to new downstream tasks. We propose a framework that pre-trains the agent in a task-agnostic manner without access to the task-specific reward. We show that our approach achieves the most diverse interacting behavior and significantly improves sample efficiency in downstream tasks.
  arXiv  Detail & Related papers  (2022-04-29T06:57:46Z)
• Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share? [7.489793155793319]
  This work focuses on combining information obtained through intrinsic motivation with the aim of having a more efficient exploration and faster learning. Our results reveal different ways in which a collaborative framework with little additional computational cost can outperform an independent learning process without knowledge sharing.
  arXiv  Detail & Related papers  (2022-02-24T16:15:51Z)
• Lipschitz-constrained Unsupervised Skill Discovery [91.51219447057817]
  Lipschitz-constrained Skill Discovery (LSD) encourages the agent to discover more diverse, dynamic, and far-reaching skills. LSD outperforms previous approaches in terms of skill diversity, state space coverage, and performance on seven downstream tasks.
  arXiv  Detail & Related papers  (2022-02-02T08:29:04Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• Open-Ended Learning Leads to Generally Capable Agents [12.079718607356178]
  We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are capable across this vast space and beyond. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours.
  arXiv  Detail & Related papers  (2021-07-27T13:30:07Z)
• Planning to Explore via Self-Supervised World Models [120.31359262226758]
  Plan2Explore is a self-supervised reinforcement learning agent. We present a new approach to self-supervised exploration and fast adaptation to new tasks. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods.
  arXiv  Detail & Related papers  (2020-05-12T17:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.