Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning

• URL: http://arxiv.org/abs/2605.13207v1
• Date: Wed, 13 May 2026 08:58:33 GMT
• Title: Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning
• Abstract summary: We introduce switching successor measures, an extension of successor measures that enables hierarchical control in reinforcement learning.<n>We show that switching successor measures arise naturally from classical successor measures while preserving their structure.<n>Experiments on both goal-conditioned and general reward-based tasks show that FB $$-Switch improves over non-hierarchical baselines.
• Score: 49.24483784910263
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hierarchical reinforcement learning can improve generalization by decomposing long-horizon decision-making into simpler subproblems. However, existing approaches often rely on restrictive design choices, such as fixed temporal abstractions or goal-conditioned objectives, which largely confine them to goal-reaching tasks and limit their applicability to general reward functions. In this paper, we introduce switching successor measures, an extension of successor measures that enables hierarchical control in zero-shot reinforcement learning without additional supervision, fixed horizons, or manually designed subgoals. We show that switching successor measures arise naturally from classical successor measures while preserving their underlying structure. Building on this result, we propose FB $π$-Switch, an algorithm that extracts both a high-level subgoal-selection policy and a low-level control policy directly from forward-backward (FB) representations, allowing hierarchical behavior to emerge from a single learned representation. Experiments on both goal-conditioned and general reward-based tasks show that FB $π$-Switch improves over non-hierarchical baselines and matches state-of-the-art hierarchical methods in goal-conditioned settings. These results demonstrate that structured successor representations provide a flexible foundation for hierarchical zero-shot reinforcement learning beyond goal-reaching tasks. Our project website is available at: https://stestokth.github.io/switching-successors/.

Related papers

• GHOST: Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation [54.305781492241785]
  We present a framework for learning visuomotor manipulation policies that generalize beyond the training distribution.<n>GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses from multi-view RGB-D observations.<n>Across a suite of tasks, this hierarchical factorization consistently improves performance and robustness compared to a flat Diffusion Policy.
  arXiv  Detail & Related papers  (2026-06-08T18:08:01Z)
• PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations [66.94988600664574]
  We present textbfPRTS (textbfPrimitive textbfReasoning and textbfTasking textbfSystem), a VLA foundation model that reformulates pretraining through Goal-Conditioned Reinforcement Learning.
  arXiv  Detail & Related papers  (2026-04-30T06:14:02Z)
• Iterative On-Policy Refinement of Hierarchical Diffusion Policies for Language-Conditioned Manipulation [6.115550515200226]
  HD-ExpIt is a framework for iterative fine-tuning of hierarchical diffusion policies via environment feedback.<n>It significantly improves hierarchical policies trained solely on offline data.
  arXiv  Detail & Related papers  (2026-03-05T15:34:43Z)
• Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL [25.40364932514488]
  We propose a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling.<n>CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks.
  arXiv  Detail & Related papers  (2026-02-03T11:11:03Z)
• Solving Sokoban using Hierarchical Reinforcement Learning with Landmarks [0.0]
  We introduce a novel hierarchical reinforcement learning framework, successfully applied to the puzzle game Sokoban.<n>Our approach constructs a six-level policy hierarchy, where each higher-level policy generates subgoals for the level below.<n>All subgoals and policies are learned end-to-end from scratch, without any domain knowledge.
  arXiv  Detail & Related papers  (2025-04-06T05:30:21Z)
• Reinforcement Learning with Options and State Representation [105.82346211739433]
  This thesis aims to explore the reinforcement learning field and build on existing methods to produce improved ones. It addresses such goals by decomposing learning tasks in a hierarchical fashion known as Hierarchical Reinforcement Learning.
  arXiv  Detail & Related papers  (2024-03-16T08:30:55Z)
• Foundation Policies with Hilbert Representations [54.44869979017766]
  We propose an unsupervised framework to pre-train generalist policies from unlabeled offline data. Our key insight is to learn a structured representation that preserves the temporal structure of the underlying environment. Our experiments show that our unsupervised policies can solve goal-conditioned and general RL tasks in a zero-shot fashion.
  arXiv  Detail & Related papers  (2024-02-23T19:09:10Z)
• Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control [44.77500987121531]
  Hierarchical Imitation Learning (HIL) has been proposed to recover highly-complex behaviors in long-horizon tasks from expert demonstrations. We develop a novel HIL algorithm based on Adversarial Inverse Reinforcement Learning. We also propose a Variational Autoencoder framework for learning with our objectives in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-10-05T00:28:26Z)
• Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space [76.46113138484947]
  General-purpose robots require diverse repertoires of behaviors to complete challenging tasks in real-world unstructured environments. To address this issue, goal-conditioned reinforcement learning aims to acquire policies that can reach goals for a wide range of tasks on command. We propose Planning to Practice, a method that makes it practical to train goal-conditioned policies for long-horizon tasks.
  arXiv  Detail & Related papers  (2022-05-17T06:58:17Z)
• On Credit Assignment in Hierarchical Reinforcement Learning [0.0]
  Hierarchical Reinforcement Learning (HRL) has held longstanding promise to advance reinforcement learning. We show how e.g., a 1-step hierarchical backup' can be seen as a conventional multistep backup with $n$ skip connections over time. We develop a new hierarchical algorithm Hier$Q_k(lambda)$, for which we demonstrate that hierarchical credit assignment alone can already boost agent performance.
  arXiv  Detail & Related papers  (2022-03-07T11:13:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.