ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning

• URL: http://arxiv.org/abs/2603.16060v2
• Date: Wed, 18 Mar 2026 20:32:14 GMT
• Title: ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning
• Abstract summary: ARISE (Agent Reasoning via Intrinsic Skill Evolution) is a hierarchical reinforcement learning framework.<n>It uses a shared policy to manage skills at high-level and to generate responses at low-level.<n>A hierarchical reward design guides the co-evolution of reasoning ability and library quality.
• Score: 17.98540130851038
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The dominant paradigm for improving mathematical reasoning in language models relies on Reinforcement Learning with verifiable rewards. Yet existing methods treat each problem instance in isolation without leveraging the reusable strategies that emerge and accumulate during training. To this end, we introduce ARISE (Agent Reasoning via Intrinsic Skill Evolution), a hierarchical reinforcement learning framework, in which a shared policy operates both to manage skills at high-level and to generate responses at low-level (denoted as a Skills Manager and a Worker, respectively). The Manager maintains a tiered skill library through a dedicated skill generation rollout that performs structured summarization of successful solution traces (after execution), while employing a policy-driven selection mechanism to retrieve relevant skills to condition future rollouts (before execution). A hierarchical reward design guides the co-evolution of reasoning ability and library quality. Experiments on two base models and seven benchmarks spanning both competition mathematics and Omni-MATH show that ARISE consistently outperforms GRPO-family algorithms and memory-augmented baselines, with particularly notable gains on out-of-distribution tasks. Ablation studies confirm that each component contributes to the observed improvements and that library quality and reasoning performance improve in tandem throughout training. Code is available at \href{https://github.com/Skylanding/ARISE}{https://github.com/Skylanding/ARISE}.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents [67.97365535829098]
  Learning from experience is critical for developing capable, self-improving large language model (LLM) agents.<n>We propose Evo-Harness, which distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation.<n>Our analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly.
  arXiv  Detail & Related papers  (2026-08-15T06:43:56Z)
• SPyCE: Skill-Policy Co-evolution for Multimodal Agents [12.105041985483167]
  We propose SPyCE, a framework that distills trajectories into a hierarchical skill library and updates it throughout reinforcement learning.<n>During training, the policy model conditions on retrieved skills to guide its rollouts, while the skill library evolves using valuable rollouts generated by the policy.<n> Experiments across eight benchmarks demonstrate that SPyCE consistently outperforms both RL-based and memory-based baselines.
  arXiv  Detail & Related papers  (2026-07-15T14:01:48Z)
• Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents [9.144810798419975]
  LLM agents increasingly retrieve externally curated skills-procedural instructions retrieved at decision time to improve performance on long-horizon interactive tasks.<n>We propose MASA Model-Aware Skill Alignment, a framework that adapts skills to each target backbone without modifying agent weights.<n> MASA consistently achieves the best overall performance, with gains of up to 25.8 points over the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-29T01:34:42Z)
• Skill-R1: Agent Skill Evolution via Reinforcement Learning [84.35984979949502]
  Skill-R1 is a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards.<n>It preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates.
  arXiv  Detail & Related papers  (2026-05-10T06:19:15Z)
• SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning [83.98129545309277]
  We propose SkillRL, a framework that bridges the gap between raw experience and policy improvement.<n>Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank.<n> Experimental results on ALF, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-09T03:17:17Z)
• Reinforcement Learning for Self-Improving Agent with Skill Library [14.717149089634718]
  Large Language Model (LLM)-based agents have demonstrated remarkable capabilities in complex reasoning and multi-turn interactions.<n>One promising approach is implementing skill libraries that allow agents to learn, validate, and apply new skills.<n>We propose a Reinforcement Learning (RL)-based approach to enhance agents' self-improvement capabilities with a skill library.
  arXiv  Detail & Related papers  (2025-12-18T21:58:19Z)
• Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling [66.0871543682453]
  We present Omni-Thinker, a unified reinforcement learning framework that scales large language models across diverse tasks.<n>Our scheduler orders tasks according to accuracy backward transfer (BWT), reducing forgetting and improving multi-task performance.
  arXiv  Detail & Related papers  (2025-07-20T01:50:16Z)
• SoftPipe: A Soft-Guided Reinforcement Learning Framework for Automated Data Preparation [10.764970149373845]
  We introduce SoftPipe, a novel RL framework that replaces rigid hard constraints'' with a flexible soft guidance'' paradigm.<n>We demonstrate that SoftPipe achieves up to a 13.9% improvement in pipeline quality and 2.8$times$ faster convergence compared to existing methods.
  arXiv  Detail & Related papers  (2025-07-18T07:43:22Z)
• Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence [131.41894248194995]
  We propose context-oriented decomposition adaptation (CorDA), a novel method that initializes adapters in a task-aware manner.<n>Thanks to the task awareness, our method enables two optional adaptation modes, knowledge-preserved mode (KPM) and instruction-previewed mode (IPM)
  arXiv  Detail & Related papers  (2025-06-16T07:55:14Z)
• Successor Feature Neural Episodic Control [17.706998080391635]
  A longstanding goal in reinforcement learning is to build intelligent agents that show fast learning and a flexible transfer of skills akin to humans and animals. This paper investigates the integration of two frameworks for tackling those goals: episodic control and successor features.
  arXiv  Detail & Related papers  (2021-11-04T19:14:43Z)
• Text Generation with Efficient (Soft) Q-Learning [91.47743595382758]
  Reinforcement learning (RL) offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. We introduce a new RL formulation for text generation from the soft Q-learning perspective. We apply the approach to a wide range of tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
  arXiv  Detail & Related papers  (2021-06-14T18:48:40Z)
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning [102.78958681141577]
  We present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy deep reinforcement learning algorithms. SUNRISE integrates two key ingredients: (a) ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble, and (b) an inference method that selects actions using the highest upper-confidence bounds for efficient exploration.
  arXiv  Detail & Related papers  (2020-07-09T17:08:44Z)
• Hierarchical Reinforcement Learning as a Model of Human Task Interleaving [60.95424607008241]
  We develop a hierarchical model of supervisory control driven by reinforcement learning. The model reproduces known empirical effects of task interleaving. The results support hierarchical RL as a plausible model of task interleaving.
  arXiv  Detail & Related papers  (2020-01-04T17:53:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.