Search Inspired Exploration in Reinforcement Learning

• URL: http://arxiv.org/abs/2602.00460v1
• Date: Sat, 31 Jan 2026 02:24:22 GMT
• Title: Search Inspired Exploration in Reinforcement Learning
• Authors: Georgios Sotirchos, Zlatan Ajanović, Jens Kober,
• Abstract summary: We propose a novel method that actively guides exploration by setting sub-goals based on the agent's learning progress.<n>Inspired by search, sub-goals are prioritized from the frontier based on estimates of cost-to-come and cost-to-go.<n>In experiments on challenging sparse-reward environments, SIERL outperforms dominant baselines in both achieving the main task goal and generalizing to reach arbitrary states in the environment.
• Score: 5.411688702405822
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Exploration in environments with sparse rewards remains a fundamental challenge in reinforcement learning (RL). Existing approaches such as curriculum learning and Go-Explore often rely on hand-crafted heuristics, while curiosity-driven methods risk converging to suboptimal policies. We propose Search-Inspired Exploration in Reinforcement Learning (SIERL), a novel method that actively guides exploration by setting sub-goals based on the agent's learning progress. At the beginning of each episode, SIERL chooses a sub-goal from the \textit{frontier} (the boundary of the agent's known state space), before the agent continues exploring toward the main task objective. The key contribution of our method is the sub-goal selection mechanism, which provides state-action pairs that are neither overly familiar nor completely novel. Thus, it assures that the frontier is expanded systematically and that the agent is capable of reaching any state within it. Inspired by search, sub-goals are prioritized from the frontier based on estimates of cost-to-come and cost-to-go, effectively steering exploration towards the most informative regions. In experiments on challenging sparse-reward environments, SIERL outperforms dominant baselines in both achieving the main task goal and generalizing to reach arbitrary states in the environment.

Related papers

• Diversity-Incentivized Exploration for Versatile Reasoning [63.653348177250756]
  We propose textbfDIVER (textbfDi-textbfIncentivized Exploration for textbfVersatiltextbfE textbfReasoning), an innovative framework that highlights the pivotal role of global sequence-level diversity to incentivize deep exploration for versatile reasoning.
  arXiv  Detail & Related papers  (2025-09-30T13:11:46Z)
• Curriculum-Based Multi-Tier Semantic Exploration via Deep Reinforcement Learning [1.8374319565577155]
  This paper presents a novel Deep Reinforcement Learning architecture that is specifically designed for resource efficient semantic exploration.<n>A key methodological contribution is the integration of a Vision-Language Model (VLM) common-sense through a layered reward function.<n>We show that our agent achieves significantly enhanced object discovery rates and develops a learned capability to effectively navigate towards semantically rich regions.
  arXiv  Detail & Related papers  (2025-09-11T11:10:08Z)
• Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning [6.266160051617362]
  "Cluster Edge Exploration" ($CE2$) is a new goal-directed exploration algorithm that gives priority to goal states that remain accessible to the agent. In challenging robotics environments, $CE2$ demonstrates superior efficiency in exploration compared to baseline methods and ablations.
  arXiv  Detail & Related papers  (2024-11-03T01:21:43Z)
• Random Latent Exploration for Deep Reinforcement Learning [71.88709402926415]
  We introduce Random Latent Exploration (RLE), a simple yet effective exploration strategy in reinforcement learning (RL)<n>On average, RLE outperforms noise-based methods, which perturb the agent's actions, and bonus-based exploration, which rewards the agent for attempting novel behaviors.<n>RLE is as simple as noise-based methods, as it avoids complex bonus calculations but retains the deep exploration benefits of bonus-based methods.
  arXiv  Detail & Related papers  (2024-07-18T17:55:22Z)
• Curricular Subgoals for Inverse Reinforcement Learning [21.038691420095525]
  Inverse Reinforcement Learning (IRL) aims to reconstruct the reward function from expert demonstrations to facilitate policy learning. Existing IRL methods mainly focus on learning global reward functions to minimize the trajectory difference between the imitator and the expert. We propose a novel Curricular Subgoal-based Inverse Reinforcement Learning framework, that explicitly disentangles one task with several local subgoals to guide agent imitation.
  arXiv  Detail & Related papers  (2023-06-14T04:06:41Z)
• On the Importance of Exploration for Generalization in Reinforcement Learning [89.63074327328765]
  We propose EDE: Exploration via Distributional Ensemble, a method that encourages exploration of states with high uncertainty. Our algorithm is the first value-based approach to achieve state-of-the-art on both Procgen and Crafter.
  arXiv  Detail & Related papers  (2023-06-08T18:07:02Z)
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning [64.97599673479678]
  We present HIerarchical reinforcement learning Guided by Landmarks (HIGL) HIGL is a novel framework for training a high-level policy with a reduced action space guided by landmarks. Our experiments demonstrate that our framework outperforms prior-arts across a variety of control tasks.
  arXiv  Detail & Related papers  (2021-10-26T12:16:19Z)
• Long-Term Exploration in Persistent MDPs [68.8204255655161]
  We propose an exploration method called Rollback-Explore (RbExplore) In this paper, we propose an exploration method called Rollback-Explore (RbExplore), which utilizes the concept of the persistent Markov decision process. We test our algorithm in the hard-exploration Prince of Persia game, without rewards and domain knowledge.
  arXiv  Detail & Related papers  (2021-09-21T13:47:04Z)
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration [52.48362697163477]
  Experimental evaluation shows that our meta-RL method significantly outperforms state-of-the-art baselines on sparse-reward tasks. We model an exploration policy learning problem for meta-RL, which is separated from exploitation policy learning. We develop a new off-policy meta-RL framework, which efficiently learns separate context-aware exploration and exploitation policies.
  arXiv  Detail & Related papers  (2020-06-15T06:56:18Z)
• LEAF: Latent Exploration Along the Frontier [47.304858727365094]
  Self-supervised goal proposal and reaching is a key component for exploration and efficient policy learning algorithms. We propose an exploration framework, which learns a dynamics-aware manifold of reachable states. We demonstrate that the proposed self-supervised exploration algorithm, superior performance compared to existing baselines on a set of challenging robotic environments.
  arXiv  Detail & Related papers  (2020-05-21T22:46:31Z)
• Intrinsic Exploration as Multi-Objective RL [29.124322674133]
  Intrinsic motivation enables reinforcement learning (RL) agents to explore when rewards are very sparse. We propose a framework based on multi-objective RL where both exploration and exploitation are being optimized as separate objectives. This formulation brings the balance between exploration and exploitation at a policy level, resulting in advantages over traditional methods.
  arXiv  Detail & Related papers  (2020-04-06T02:37:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.