CODE-SHARP: Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs

• URL: http://arxiv.org/abs/2602.10085v2
• Date: Wed, 11 Feb 2026 09:46:16 GMT
• Title: CODE-SHARP: Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs
• Authors: Richard Bornemann, Pierluigi Vito Amadori, Antoine Cully,
• Abstract summary: We introduce Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs (CODE-SHARP)<n>We show that a goal-conditioned agent trained exclusively on the rewards generated by the discovered skills learns to solve increasingly long-horizon goals.<n>When composed by a high-level FM-based planner, the discovered skills enable a single goal-conditioned agent to solve complex, long-horizon tasks, outperforming both pretrained agents and task-specific expert policies by over $134$% on average.
• Score: 8.81909423168606
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Developing agents capable of open-endedly discovering and learning novel skills is a grand challenge in Artificial Intelligence. While reinforcement learning offers a powerful framework for training agents to master complex skills, it typically relies on hand-designed reward functions. This is infeasible for open-ended skill discovery, where the set of meaningful skills is not known a priori. While recent methods have shown promising results towards automating reward function design, they remain limited to refining rewards for pre-defined tasks. To address this limitation, we introduce Continuous Open-ended Discovery and Evolution of Skills as Hierarchical Reward Programs (CODE-SHARP), a novel framework leveraging Foundation Models (FM) to open-endedly expand and refine a hierarchical skill archive, structured as a directed graph of executable reward functions in code. We show that a goal-conditioned agent trained exclusively on the rewards generated by the discovered SHARP skills learns to solve increasingly long-horizon goals in the Craftax environment. When composed by a high-level FM-based planner, the discovered skills enable a single goal-conditioned agent to solve complex, long-horizon tasks, outperforming both pretrained agents and task-specific expert policies by over $134$% on average. We will open-source our code and provide additional videos at https://sites.google.com/view/code-sharp/homepage.

Related papers

• SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning [83.98129545309277]
  We propose SkillRL, a framework that bridges the gap between raw experience and policy improvement.<n>Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank.<n> Experimental results on ALF, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-09T03:17:17Z)
• Dreaming in Code for Curriculum Learning in Open-Ended Worlds [11.954246951892905]
  Dreaming in Code (DiCode) is a framework in which foundation models synthesize environment code to learn toward increasing competence.<n>We instantiate DiCode in Craftax, a challenging open-ended benchmark characterized by rich mechanics and longhorizon progression.<n>Our results suggest that code-level environment design provides a practical mechanism for curriculum control, enabling the construction of intermediate environments that bridge competence gaps in open-ended worlds.
  arXiv  Detail & Related papers  (2026-02-09T01:24:40Z)
• Agentic Skill Discovery [19.5703917813767]
  Language-conditioned robotic skills make it possible to apply the high-level reasoning of Large Language Models (LLMs) to low-level robotic control. A remaining challenge is to acquire a diverse set of fundamental skills. We introduce a novel framework for skill discovery that is entirely driven by LLMs.
  arXiv  Detail & Related papers  (2024-05-23T19:44:03Z)
• Eureka: Human-Level Reward Design via Coding Large Language Models [121.91007140014982]
  Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. We present Eureka, a human-level reward design algorithm powered by LLMs. Eureka exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2023-10-19T17:31:01Z)
• Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidance [66.615355754712]
  BOSS learns to accomplish new tasks by performing "skill bootstrapping" We demonstrate through experiments in realistic household environments that agents trained with our LLM-guided bootstrapping procedure outperform those trained with naive bootstrapping.
  arXiv  Detail & Related papers  (2023-10-16T02:43:47Z)
• Skill-Based Reinforcement Learning with Intrinsic Reward Matching [77.34726150561087]
  We present Intrinsic Reward Matching (IRM), which unifies task-agnostic skill pretraining and task-aware finetuning. IRM enables us to utilize pretrained skills far more effectively than previous skill selection methods.
  arXiv  Detail & Related papers  (2022-10-14T00:04:49Z)
• Open-Ended Reinforcement Learning with Neural Reward Functions [2.4366811507669115]
  In high-dimensional robotic environments our approach learns a wide range of interesting skills including front-flips for Half-Cheetah and one-legged running for Humanoid. In the pixel-based Montezuma's Revenge environment our method also works with minimal changes and it learns complex skills that involve interacting with items and visiting diverse locations.
  arXiv  Detail & Related papers  (2022-02-16T15:55:22Z)
• Lipschitz-constrained Unsupervised Skill Discovery [91.51219447057817]
  Lipschitz-constrained Skill Discovery (LSD) encourages the agent to discover more diverse, dynamic, and far-reaching skills. LSD outperforms previous approaches in terms of skill diversity, state space coverage, and performance on seven downstream tasks.
  arXiv  Detail & Related papers  (2022-02-02T08:29:04Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• ELSIM: End-to-end learning of reusable skills through intrinsic motivation [0.0]
  We present a novel reinforcement learning architecture which hierarchically learns and represents self-generated skills in an end-to-end way. With this architecture, an agent focuses only on task-rewarded skills while keeping the learning process of skills bottom-up.
  arXiv  Detail & Related papers  (2020-06-23T11:20:46Z)
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning [81.12201426668894]
  We develop efficient reinforcement learning methods that acquire diverse skills without any reward function, and then repurpose these skills for downstream tasks. We show that our proposed algorithm provides substantial improvement in learning efficiency, making reward-free real-world training feasible. We also demonstrate that the learned skills can be composed using model predictive control for goal-oriented navigation, without any additional training.
  arXiv  Detail & Related papers  (2020-04-27T17:38:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.