APEX: Autonomous Policy Exploration for Self-Evolving LLM Agents

• URL: http://arxiv.org/abs/2605.21240v1
• Date: Wed, 20 May 2026 14:29:27 GMT
• Title: APEX: Autonomous Policy Exploration for Self-Evolving LLM Agents
• Abstract summary: Self-evolving agents learn by accumulating memory and reflection across episodes rather than requiring model-weight updates.<n>As memory grows, behavior concentrates around familiar high-reward routines, reducing the chance of discovering better alternatives.<n>We propose Autonomous Policy EXploration (APEX), which builds and maintains an explicit strategy space through a strategy map.
• Score: 54.213455157510445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents have shown strong performance across a wide range of complex tasks, including interactive environments that require long-horizon decision making. But these agents cannot learn on the fly at test time. Self-evolving agents address this by accumulating memory and reflection across episodes rather than requiring model-weight updates. However, these agents often suffer from exploration collapse: as memory grows, behavior concentrates around familiar high-reward routines, reducing the chance of discovering better alternatives. To address this problem, we propose Autonomous Policy EXploration (APEX), which builds and maintains an explicit strategy space through a strategy map-a directed acyclic graph of milestones with prerequisite dependency edges. In APEX, Fork Discovery expands the map with evidence-grounded unexplored directions, while Policy Selection balances exploration and exploitation during planning. Evaluated on nine Jericho text-adventure games and WebArena, a realistic web interaction benchmark, APEX outperforms all baselines. Extensive ablations validate each component's contribution and demonstrate robustness across diverse settings, demonstrating APEX's effectiveness for sustained exploration in self-evolving agents.

Related papers

• Multi-Agent LLMs Fail to Explore Each Other [34.74439361133657]
  We show that modern large language model (LLM) agents fail to explore effectively when interacting with one another.<n>We introduce Multi- Agent Contextual Exploration (MACE), a lightweight framework that explicitly promotes exploration through structured peer selection.
  arXiv  Detail & Related papers  (2026-07-13T08:34:05Z)
• Learning to Adapt: Self-Improving Web Agent via Cognitive-Aware Exploration [41.778518335766215]
  We propose a framework for building truly autonomous and adaptive web agents.<n>We use three adversarial roles, Selector, Predictor, and Judger, to discover the agent's limitations and expand its cognitive boundaries.<n>Our framework offers a scalable and generalizable solution for building truly autonomous and adaptive web agents.
  arXiv  Detail & Related papers  (2026-05-29T14:37:27Z)
• Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization [14.566152113959555]
  A key limitation of existing methods is that they typically employ undifferentiated exploration strategies.<n>We propose an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high.
  arXiv  Detail & Related papers  (2026-05-09T14:44:18Z)
• AIPO: Learning to Reason from Active Interaction [54.10819421625103]
  AIPO enables the policy model to proactively consult three functional collaborative agents when encountering reasoning bottlenecks.<n>AIPO consistently improves reasoning performance, generalizes robustly across different policy models and RLVR algorithms, and effectively expands the reasoning capability boundary of the policy model.
  arXiv  Detail & Related papers  (2026-05-08T19:06:55Z)
• Your Agent is More Brittle Than You Think: Uncovering Indirect Injection Vulnerabilities in Agentic LLMs [32.38053469964495]
  Extended action spaces, including hidden inter-system interactions, pose severe security challenges.<n>IPI, which conceals malicious instructions within third-party content, can trigger unauthorized actions such as data exfiltration.<n>We evaluate six defense strategies against four sophisticated IPI attack vectors across nine LLM backbones.
  arXiv  Detail & Related papers  (2026-04-04T21:27:04Z)
• OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions [66.84396313837765]
  We introduce OdysseyArena, which re-centers agent evaluation on long-horizon, active, and inductive interactions.<n>We provide a set of 120 tasks to measure an agent's inductive efficiency and long-horizon discovery.<n>We also introduce OdysseyArena-Challenge to stress-test agent stability across extreme interaction horizons.
  arXiv  Detail & Related papers  (2026-02-05T16:31:43Z)
• AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts [78.33143446024485]
  We introduce textbfAgentLongBench, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles.<n>This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios.
  arXiv  Detail & Related papers  (2026-01-28T16:05:44Z)
• IterResearch: Rethinking Long-Horizon Agents via Markovian State Reconstruction [107.49922328855025]
  IterResearch is a novel iterative deep-research paradigm that reformulates long-horizon research as a Markov Decision Process.<n>It achieves substantial improvements over existing open-source agents with average +14.5pp across six benchmarks.<n>It serves as an effective prompting strategy, improving frontier models by up to 19.2pp over ReAct on long-horizon tasks.
  arXiv  Detail & Related papers  (2025-11-10T17:30:08Z)
• The Landscape of Agentic Reinforcement Learning for LLMs: A Survey [103.32591749156416]
  The emergence of agentic reinforcement learning (Agentic RL) marks a paradigm shift from conventional reinforcement learning applied to large language models (LLM RL)<n>This survey formalizes this conceptual shift by contrasting the degenerate single-step Markov Decision Processes (MDPs) of LLM-RL with the temporally extended, partially observable Markov decision processes (POMDPs) that define Agentic RL.
  arXiv  Detail & Related papers  (2025-09-02T17:46:26Z)
• Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning [27.81925751697255]
  We propose a novel method for efficient multi-agent exploration in complex scenarios. We formulate the imagination as a sequence modeling problem, where the states, observations, prompts, actions, and rewards are predicted autoregressively. By initializing agents at the critical states, IIE significantly increases the likelihood of discovering potentially important underexplored regions.
  arXiv  Detail & Related papers  (2024-02-28T01:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.