MRSD: Multi-Resolution Skill Discovery for HRL Agents

• URL: http://arxiv.org/abs/2505.21410v1
• Date: Tue, 27 May 2025 16:38:55 GMT
• Title: MRSD: Multi-Resolution Skill Discovery for HRL Agents
• Authors: Shashank Sharma, Janina Hoffmann, Vinay Namboodiri,
• Abstract summary: We propose Multi-Resolution Skill Discovery (MRSD), an HRL framework that learns multiple skill encoders at different temporal resolutions in parallel.<n>A high-level manager dynamically selects among these skills, enabling adaptive control strategies over time.<n>MRSD outperforms prior state-of-the-art skill discovery and HRL methods, achieving faster convergence and higher final performance.
• Score: 2.1438108757511958
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hierarchical reinforcement learning (HRL) relies on abstract skills to solve long-horizon tasks efficiently. While existing skill discovery methods learns these skills automatically, they are limited to a single skill per task. In contrast, humans learn and use both fine-grained and coarse motor skills simultaneously. Inspired by human motor control, we propose Multi-Resolution Skill Discovery (MRSD), an HRL framework that learns multiple skill encoders at different temporal resolutions in parallel. A high-level manager dynamically selects among these skills, enabling adaptive control strategies over time. We evaluate MRSD on tasks from the DeepMind Control Suite and show that it outperforms prior state-of-the-art skill discovery and HRL methods, achieving faster convergence and higher final performance. Our findings highlight the benefits of integrating multi-resolution skills in HRL, paving the way for more versatile and efficient agents.

Related papers

• EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data [22.471559284344462]
  Most reinforcement learning (RL) methods focus on learning optimal policies over low-level action spaces. While these methods can perform well in their training environments, they lack the flexibility to transfer to new tasks. We demonstrate through experiments in sparse, image-based, robot manipulation environments that can more quickly learn new tasks than prior works.
  arXiv  Detail & Related papers  (2024-06-25T17:50:03Z)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL [80.10358123795946]
  We develop a framework for building multi-turn RL algorithms for fine-tuning large language models. Our framework adopts a hierarchical RL approach and runs two RL algorithms in parallel. Empirically, we find that ArCHer significantly improves efficiency and performance on agent tasks.
  arXiv  Detail & Related papers  (2024-02-29T18:45:56Z)
• Primitive Skill-based Robot Learning from Human Evaluative Feedback [28.046559859978597]
  Reinforcement learning algorithms face challenges when dealing with long-horizon robot manipulation tasks in real-world environments. We propose a novel framework, SEED, which leverages two approaches: reinforcement learning from human feedback (RLHF) and primitive skill-based reinforcement learning. Our results show that SEED significantly outperforms state-of-the-art RL algorithms in sample efficiency and safety.
  arXiv  Detail & Related papers  (2023-07-28T20:48:30Z)
• Granger Causal Interaction Skill Chains [35.143372688036685]
  Reinforcement Learning (RL) has demonstrated promising results in learning policies for complex tasks, but it often suffers from low sample efficiency and limited transferability. We introduce the Chain of Interaction Skills (COInS) algorithm, which focuses on controllability factored in domains to identify a small number of task-agnostic skills that still permit a high degree of control. We also demonstrate the transferability of skills learned by COInS, using variants of Breakout, a common RL benchmark, and show 2-3x improvement in both sample efficiency and final performance compared to standard RL baselines.
  arXiv  Detail & Related papers  (2023-06-15T21:06:54Z)
• Controllability-Aware Unsupervised Skill Discovery [94.19932297743439]
  We introduce a novel unsupervised skill discovery method, Controllability-aware Skill Discovery (CSD), which actively seeks complex, hard-to-control skills without supervision. The key component of CSD is a controllability-aware distance function, which assigns larger values to state transitions that are harder to achieve with the current skills. Our experimental results in six robotic manipulation and locomotion environments demonstrate that CSD can discover diverse complex skills with no supervision.
  arXiv  Detail & Related papers  (2023-02-10T08:03:09Z)
• Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics [18.546688182454236]
  Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. We propose accelerating exploration in the skill space using state-conditioned generative models. We validate our approach across four challenging manipulation tasks, demonstrating our ability to learn across task variations.
  arXiv  Detail & Related papers  (2022-11-04T02:42:17Z)
• Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery [12.586875201983778]
  Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for training neural policies to solve complex control tasks. We show that Quality Diversity (QD) methods are a competitive alternative to information-theory-augmented RL for skill discovery.
  arXiv  Detail & Related papers  (2022-10-06T11:06:39Z)
• Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning [73.92475751508452]
  Bimanual Dexterous Hands Benchmark (Bi-DexHands) is a simulator that involves two dexterous hands with tens of bimanual manipulation tasks and thousands of target objects. Tasks in Bi-DexHands are designed to match different levels of human motor skills according to cognitive science literature.
  arXiv  Detail & Related papers  (2022-06-17T11:09:06Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks [85.56153200251713]
  We introduce EMBR, a model-based RL method for learning primitive skills that are suitable for completing long-horizon visuomotor tasks. On a Franka Emika robot arm, we find that EMBR enables the robot to complete three long-horizon visuomotor tasks at 85% success rate.
  arXiv  Detail & Related papers  (2021-09-21T16:48:07Z)
• Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control [65.00425082663146]
  We present a Knowledge Transfer based Multi-task Deep Reinforcement Learning framework (KTM-DRL) for continuous control. In KTM-DRL, the multi-task agent first leverages an offline knowledge transfer algorithm to quickly learn a control policy from the experience of task-specific teachers. The experimental results well justify the effectiveness of KTM-DRL and its knowledge transfer and online learning algorithms, as well as its superiority over the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2020-10-15T03:26:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.