Multi-level meta-reinforcement learning with skill-based curriculum

• URL: http://arxiv.org/abs/2603.08773v1
• Date: Mon, 09 Mar 2026 17:59:39 GMT
• Title: Multi-level meta-reinforcement learning with skill-based curriculum
• Abstract summary: We describe an efficient multi-level procedure for repeatedly compressing Markov decision processes (MDPs)<n>A parametric family of policies at one level is treated as single actions in the compressed MDPs at higher levels, while preserving the semantic meanings and structure of the original MDP.<n>A second fundamental aspect of this work is that these multi-level decompositions plus the factorization of policies yield new transfer opportunities of skills across different problems and different levels.
• Score: 2.728639503114606
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider problems in sequential decision making with natural multi-level structure, where sub-tasks are assembled together to accomplish complex goals. Systematically inferring and leveraging hierarchical structure has remained a longstanding challenge; we describe an efficient multi-level procedure for repeatedly compressing Markov decision processes (MDPs), wherein a parametric family of policies at one level is treated as single actions in the compressed MDPs at higher levels, while preserving the semantic meanings and structure of the original MDP, and mimicking the natural logic to address a complex MDP. Higher-level MDPs are themselves independent MDPs with less stochasticity, and may be solved using existing algorithms. As a byproduct, spatial or temporal scales may be coarsened at higher levels, making it more efficient to find long-term optimal policies. The multi-level representation delivered by this procedure decouples sub-tasks from each other and usually greatly reduces unnecessary stochasticity and the policy search space, leading to fewer iterations and computations when solving the MDPs. A second fundamental aspect of this work is that these multi-level decompositions plus the factorization of policies into embeddings (problem-specific) and skills (including higher-order functions) yield new transfer opportunities of skills across different problems and different levels. This whole process is framed within curriculum learning, wherein a teacher organizes the student agent's learning process in a way that gradually increases the difficulty of tasks and and promotes transfer across MDPs and levels within and across curricula. The consistency of this framework and its benefits can be guaranteed under mild assumptions. We demonstrate abstraction, transferability, and curriculum learning in examples, including MazeBase+, a more complex variant of the MazeBase example.

Related papers

• Sequence Pathfinder for Multi-Agent Pickup and Delivery in the Warehouse [10.576983033957953]
  Multi-Agent Pickup and Delivery (MAPD) is a challenging extension of Multi-Agent Path Finding (MAPF)<n> Communication learning can alleviate the lack of global information but introduce high computational complexity due to point-to-point communication.<n>We propose the Sequential Pathfinder (SePar) to achieve implicit information exchange, reducing decision-making complexity from exponential to linear.
  arXiv  Detail & Related papers  (2025-09-28T09:48:13Z)
• Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents [56.625878022978945]
  Large Language Models (LLMs) as autonomous agents are increasingly tasked with solving complex, long-horizon problems.<n>Direct Preference Optimization (DPO) provides a signal that is too coarse for precise credit assignment, while step-level DPO is often too myopic to capture the value of multi-step behaviors.<n>We introduce Hierarchical Preference Learning (HPL), a hierarchical framework that optimize LLM agents by leveraging preference signals at multiple, synergistic granularities.
  arXiv  Detail & Related papers  (2025-09-26T08:43:39Z)
• Collab-Solver: Collaborative Solving Policy Learning for Mixed-Integer Linear Programming [57.44900640134789]
  We propose a novel multi-agent-based policy learning framework for MILP solving as a Stackelberg game.<n>Specifically, we formulate the collaboration of cut selection and branching in MILP solving as a Stackelberg game.<n>The jointly learned policy significantly improves the solving performance on both synthetic and large-scale real-world MILP datasets.
  arXiv  Detail & Related papers  (2025-08-05T03:16:04Z)
• Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning [19.739901034066587]
  We propose a novel and effective methodology for learning multi-agent hierarchies of message-passing policies.<n>Agents at lower levels in the hierarchy receive goals from the upper levels and exchange messages with neighboring agents at the same level.<n>Results on relevant benchmarks show that our method performs favorably compared to the state of the art.
  arXiv  Detail & Related papers  (2025-07-31T14:42:12Z)
• Hierarchical Average-Reward Linearly-solvable Markov Decision Processes [11.69049916139847]
  We introduce a novel approach to hierarchical reinforcement learning for Linearly-solvable Markov Decision Processes. Our approach allows learning low-level and high-level tasks simultaneously, without imposing limiting restrictions on the low-level tasks. Experiments show that our approach can outperform flat average-reward reinforcement learning by one or several orders of magnitude.
  arXiv  Detail & Related papers  (2024-07-09T09:06:44Z)
• Hierarchical Decision Making Based on Structural Information Principles [19.82391136775341]
  We propose a novel Structural Information principles-based framework, namely SIDM, for hierarchical Decision Making.<n>We present an abstraction mechanism that processes historical state-action trajectories to construct abstract representations of states and actions.<n>We develop a skill-based learning method for single-agent scenarios and a role-based collaboration method for multi-agent scenarios, both of which can flexibly integrate various underlying algorithms for enhanced performance.
  arXiv  Detail & Related papers  (2024-04-15T13:02:00Z)
• Provable Benefits of Multi-task RL under Non-Markovian Decision Making Processes [56.714690083118406]
  In multi-task reinforcement learning (RL) under Markov decision processes (MDPs), the presence of shared latent structures has been shown to yield significant benefits to the sample efficiency compared to single-task RL. We investigate whether such a benefit can extend to more general sequential decision making problems, such as partially observable MDPs (POMDPs) and more general predictive state representations (PSRs) We propose a provably efficient algorithm UMT-PSR for finding near-optimal policies for all PSRs, and demonstrate that the advantage of multi-task learning manifests if the joint model class of PSR
  arXiv  Detail & Related papers  (2023-10-20T14:50:28Z)
• Intrinsically Motivated Hierarchical Policy Learning in Multi-objective Markov Decision Processes [15.50007257943931]
  We propose a novel dual-phase intrinsically motivated reinforcement learning method to address this limitation. We show experimentally that the proposed method significantly outperforms state-of-the-art multi-objective reinforcement methods in a dynamic robotics environment.
  arXiv  Detail & Related papers  (2023-08-18T02:10:45Z)
• Multi-Objective Policy Gradients with Topological Constraints [108.10241442630289]
  We present a new algorithm for a policy gradient in TMDPs by a simple extension of the proximal policy optimization (PPO) algorithm. We demonstrate this on a real-world multiple-objective navigation problem with an arbitrary ordering of objectives both in simulation and on a real robot.
  arXiv  Detail & Related papers  (2022-09-15T07:22:58Z)
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs [55.31018404591743]
  We leverage ideas of common structure from the HiP-MDP setting to enable robust state abstractions inspired by Block MDPs. We derive instantiations of this new framework for both multi-task reinforcement learning (MTRL) and meta-reinforcement learning (Meta-RL) settings.
  arXiv  Detail & Related papers  (2020-07-14T17:25:27Z)
• Hierarchical Variational Imitation Learning of Control Programs [131.7671843857375]
  We propose a variational inference method for imitation learning of a control policy represented by parametrized hierarchical procedures (PHP) Our method discovers the hierarchical structure in a dataset of observation-action traces of teacher demonstrations, by learning an approximate posterior distribution over the latent sequence of procedure calls and terminations. We demonstrate a novel benefit of variational inference in the context of hierarchical imitation learning: in decomposing the policy into simpler procedures, inference can leverage acausal information that is unused by other methods.
  arXiv  Detail & Related papers  (2019-12-29T08:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.