Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization

• URL: http://arxiv.org/abs/2605.08978v2
• Date: Tue, 12 May 2026 02:23:49 GMT
• Title: Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
• Abstract summary: A key limitation of existing methods is that they typically employ undifferentiated exploration strategies.<n>We propose an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high.
• Score: 14.566152113959555
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in agentic test-time scaling allow models to gather environmental feedback before committing to final actions. A key limitation of existing methods is that they typically employ undifferentiated exploration strategies, lacking the ability to adaptively distinguish when exploration is truly required. In this paper, we propose an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high. Our method introduces a fine-grained reward function via variational inference that explicitly evaluates exploratory actions by estimating their potential to improve future decision-making, together with an exploration-aware grouping mechanism that separates exploratory actions from task-completion actions during optimization. By targeting informational gaps, this design allows agents to explore selectively and transition to execution as soon as the task context is clear. Empirically, we demonstrate that our approach achieves consistent improvements across a range of challenging text-based and GUI-based agent benchmarks. Code is available at https://github.com/HansenHua/EAPO-ICML26 and models are available at https://huggingface.co/hansenhua/EAPO-ICML26.

Related papers

• OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents [74.20327254615854]
  Large language model agents interleave reasoning, action selection, and observation to solve sequential decision-making tasks.<n>Existing inference-time adaptation methods for LLM agents mainly rely on prompting or retrieval.<n>We propose OLIVIA, an inference-time action adaptation framework for ReAct-style agents.
  arXiv  Detail & Related papers  (2026-05-11T19:28:20Z)
• Poly-EPO: Training Exploratory Reasoning Models [62.82992914206963]
  We present a framework for post-training language models (LMs) that explicitly encourages optimistic exploration and promotes a synergy between exploration and exploitation.<n>We propose Polychromic Exploratory Policy Optimization (Poly-EPO), which instantiates this framework with an objective that explicitly synergizes exploration and exploitation.
  arXiv  Detail & Related papers  (2026-04-19T22:54:19Z)
• Exploration and Exploitation Errors Are Measurable for Language Model Agents [52.95061000593404]
  Language Model (LM) agents are increasingly used in complex openended decision tasks.<n>A core requirement is the ability to both explore the problem space and exploit acquired knowledge effectively.<n>We design controllable environments inspired by practical embodied AI scenarios.
  arXiv  Detail & Related papers  (2026-04-14T17:59:57Z)
• SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents [0.31034395048547575]
  Existing approaches rely on static prompting strategies or handcrafteds to select actions such as code editing, file navigation, and test execution.<n>We propose SWE-Shepherd, a framework that introduces Process Reward Models (PRMs) to provide dense, step-level supervision for repository-level code agents.<n>Experiments on SWE-Bench Verified demonstrate improved interaction efficiency and action quality, while also highlighting challenges in aligning intermediate rewards with final task success.
  arXiv  Detail & Related papers  (2026-04-12T06:51:47Z)
• Multi-Agent LLMs for Adaptive Acquisition in Bayesian Optimization [2.6954666679827137]
  We show how Large Language Models (LLMs) construct and adapt exploration-exploitation strategies.<n>We propose a multi-agent framework that decomposes exploration-exploitation control into strategic policy mediation and tactical candidate generation.
  arXiv  Detail & Related papers  (2026-03-30T20:05:30Z)
• Novelty-Driven Target-Space Discovery in Automated Electron and Scanning Probe Microscopy [0.5805874695844994]
  We develop a deep-learning BEACON framework to guide discovery in the target space.<n>This benchmarking framework provides a basis for evaluating discovery-driven algorithms.<n>The associated notebooks are available, allowing users to reproduce, test the benchmarks, and adapt the method to their own instruments and datasets.
  arXiv  Detail & Related papers  (2026-03-17T16:04:20Z)
• Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Exploration [52.35887679314727]
  Long-term Memory Embodied Exploration aims to unify the agent's exploratory cognition and decision-making behaviors.<n>To enhance the agent's memory recall and proactive exploration capabilities, we propose MemoryExplorer.
  arXiv  Detail & Related papers  (2026-01-11T16:23:22Z)
• AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress [71.02263260394261]
  Large language models (LLMs) still encounter challenges in multi-turn decision-making tasks.<n>We build process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process.<n>AgentPRM captures both the interdependence between sequential decisions and their contribution to the final goal.
  arXiv  Detail & Related papers  (2025-11-11T14:57:54Z)
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process.<n>We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning [27.81925751697255]
  We propose a novel method for efficient multi-agent exploration in complex scenarios. We formulate the imagination as a sequence modeling problem, where the states, observations, prompts, actions, and rewards are predicted autoregressively. By initializing agents at the critical states, IIE significantly increases the likelihood of discovering potentially important underexplored regions.
  arXiv  Detail & Related papers  (2024-02-28T01:45:01Z)
• LASER: LLM Agent with State-Space Exploration for Web Navigation [57.802977310392755]
  Large language models (LLMs) have been successfully adapted for interactive decision-making tasks like web navigation. Previous methods implicitly assume a forward-only execution mode for the model, where they only provide oracle trajectories as in-context examples. We propose to model the interactive task as state space exploration, where the LLM agent transitions among a pre-defined set of states by performing actions to complete the task.
  arXiv  Detail & Related papers  (2023-09-15T05:44:08Z)
• Sequential Transfer in Reinforcement Learning with a Generative Model [48.40219742217783]
  We show how to reduce the sample complexity for learning new tasks by transferring knowledge from previously-solved ones. We derive PAC bounds on its sample complexity which clearly demonstrate the benefits of using this kind of prior knowledge. We empirically verify our theoretical findings in simple simulated domains.
  arXiv  Detail & Related papers  (2020-07-01T19:53:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.