EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning

• URL: http://arxiv.org/abs/2608.21946v2
• Date: Wed, 26 Aug 2026 15:51:09 GMT
• Title: EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning
• Abstract summary: reusable exploration patterns embedded in interaction trajectories are largely discarded after a single policy update.<n>We propose EDGE (Experience-Distillation for Guided Exploration), a framework that treats retrieved experiences as temporary training-time scaffolds.<n> EDGE improves over strong RL baselines by up to 12.5 points and remains effective without inference-time scaffolds or a proprietary reflector.
• Score: 25.200068256200293
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning with outcome-based objectives such as GRPO enables LLM-based agents to solve complex, long-horizon tasks, yet the reusable exploration patterns embedded in interaction trajectories are largely discarded after a single policy update. Existing experience-augmented approaches retrieve historical guidance at inference time, but they apply experiences without accounting for the policy's evolving capability and create persistent dependencies on external retrieval. We propose EDGE (Experience-Distillation for Guided Exploration), a framework that treats retrieved experiences as temporary training-time scaffolds and progressively internalizes their benefits into the parametric policy. Concretely, EDGE partitions each rollout group into experience-conditioned and experience-free trajectories to estimate and admit only positive marginal gains without extra sampling, then distills the induced behavior into the base policy via a reverse-KL objective on its own empirical support. A co-evolutionary experience bank further synthesizes guidance from emerging failure modes and prunes obsolete entries as the policy evolves. Across embodied, web, and search-based QA tasks, EDGE improves over strong RL baselines by up to 12.5 points and remains effective without inference-time scaffolds or a proprietary reflector. The code is available at https://github.com/xvolcano02/EDGE.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• Self-Improving Large Language Models via Progressive Experience Evolution [52.03479747358687]
  We propose textbfSPEE (textbfSelf-textbfProgressive textbfExperience textbfEvolution), a unified post-training framework that sequentially performs explicit experience evolution followed by implicit policy optimization.<n> Experiments on five mathematical reasoning benchmarks demonstrate that SPEE consistently outperforms both test-time and training-time self-evolution baselines across three model scales.
  arXiv  Detail & Related papers  (2026-08-03T12:27:32Z)
• PATH-Bench: Path-Dependent Evaluation of Lifelong Agents [27.837334107109786]
  PATH-Bench is a benchmark for path-dependent evaluation of lifelong agents.<n>We propose Selective Experience Use (SEU), an agent harness that regulates how path-accumulated experience influences each new task.
  arXiv  Detail & Related papers  (2026-08-02T11:00:04Z)
• Group-Reflective Self-Distillation for Agentic Reinforcement Learning [8.041052886037766]
  Group-Reflective Self-Distillation (GRSD) derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts.<n>GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.
  arXiv  Detail & Related papers  (2026-07-30T11:49:46Z)
• SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning [38.54413500261524]
  Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions.<n>We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills.
  arXiv  Detail & Related papers  (2026-07-16T09:57:18Z)
• RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [76.17893114021757]
  Deep research systems that plan, search, evaluate evidence, and synthesize long-form reports lack ground-truth answers and span many tool-augmented decisions.<n>In this work, we argue that rubrics should serve not merely as final-answer evaluators, but as the shared interface that structures policy execution, judge feedback, and agent memory.<n>We introduceEM, a rubric-guided reinforcement learning framework that combines stagewise policy decomposition with reflection-based meta-policy evolution.
  arXiv  Detail & Related papers  (2026-05-11T17:40:38Z)
• AIPO: Learning to Reason from Active Interaction [54.10819421625103]
  AIPO enables the policy model to proactively consult three functional collaborative agents when encountering reasoning bottlenecks.<n>AIPO consistently improves reasoning performance, generalizes robustly across different policy models and RLVR algorithms, and effectively expands the reasoning capability boundary of the policy model.
  arXiv  Detail & Related papers  (2026-05-08T19:06:55Z)
• Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation [50.696688705287755]
  We propose Mutual Information Self-Evaluation to overcome the sparse reward challenge in reinforcement learning.<n>MISE enables an agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals.<n>We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy.
  arXiv  Detail & Related papers  (2026-04-13T15:18:51Z)
• Internalizing Agency from Reflective Experience [20.650609947690196]
  LEAFE is a framework that internalizes recovery agency from reflective experience.<n>It consistently improves Pass@1 over the base model and achieves higher Pass@k than outcome-driven baselines.
  arXiv  Detail & Related papers  (2026-03-17T17:50:47Z)
• RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback [54.39884046754265]
  RetroAgent is an online RL framework that enables agents to master complex interactive environments.<n>Our experiments show that RetroAgent achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-03-09T16:23:33Z)
• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning [41.90621652673528]
  We propose SPEAR, a curriculum-based self-imitation learning (SIL) recipe for training agentic LLMs.<n>Specifically, our approach incorporates a curriculum to manage the exploration process, utilizing intrinsic rewards to foster skill-level exploration.<n>To further stabilize training, we recalibrate the advantages of experiences in the replay buffer to address the potential policy drift.
  arXiv  Detail & Related papers  (2025-09-26T17:20:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.