OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation

• URL: http://arxiv.org/abs/2606.17628v1
• Date: Tue, 16 Jun 2026 07:33:53 GMT
• Title: OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation
• Abstract summary: Memory has become a standard substrate for self-evolving agents, yet retaining experience is not the same as learning how to evolve through it.<n>We introduce OPD-Evolver, a slow-fast co-evolution framework that cultivates such an agent evolver through on-policy self-distillation.
• Score: 71.57258488733974
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Memory has become a standard substrate for self-evolving agents, yet retaining experience is not the same as learning how to evolve through it. Existing memory agents can store trajectories, retrieve reflections, or accumulate skills, but often lack the holistic competence to select useful experience, act on it, write reusable knowledge, and maintain a growing repository. We introduce OPD-Evolver, a slow-fast co-evolution framework that cultivates such an agent evolver through on-policy self-distillation. In the fast loop, OPD-Evolver interacts with a four-level memory hierarchy to read, use, write, and maintain experience for rapid test-time evolution. In the slow loop, outcome-calibrated memory attribution and privileged hindsight distill these four abilities into the deployable policy. Across multi-domain benchmarks, OPD-Evolver surpasses memory systems such as ReasoningBank by up to 11.5%, and training-based methods such as Skill0 by ~5.8%. Further analysis shows that OPD-Evolver internalizes high-value experience and memory management, enabling OPD-Evolver-9B to challenge giant counterparts such as Qwen3.5-397B-A17B and Step-3.5-Flash, pointing beyond memory-augmented agents toward genuinely qualified agent evolvers.

Related papers

• ERSkill: Evolving for Skill-Guided Adaptive Memory Retrieval [8.099546846655196]
  ERSkill is a retrieval-centric framework for self-evolving, skill-guided memory access.<n>It co-evolves the skill set and the router during training to enable continuous improvement.<n>It substantially outperforms strong non-evolving and self-evolving baselines.
  arXiv  Detail & Related papers  (2026-08-13T02:06:01Z)
• MindMemOS: A Portable and Self-Evolving Memory Operating Layer for AI Agents [31.48763155266752]
  Memory is a core component of AI agents, enabling them to accumulate experience, maintain personalization, and adapt over long-term interactions.<n>We present MindMemOS, a portable and self-evolving memory operating layer that organizes open-world information using a unified entity property timestructure.<n>MindMemOS supports scenario-adaptive memory modeling, higher-order pattern discovery, autonomous memory refinement, and continuous skill evolution.
  arXiv  Detail & Related papers  (2026-08-12T11:29:49Z)
• Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember [26.754732469136385]
  Self-play agents can generate training problems without questions from target benchmarks, but their curricula lack persistent state.<n>We introduce textbfSESA (Self-Evolving Skill-Augmented Agent), which makes procedural memory an evolving state of tool-augmented search self-play.
  arXiv  Detail & Related papers  (2026-07-31T14:32:52Z)
• Dynamic Mixture of Latent Memories for Self-Evolving Agents [57.20419419302731]
  MoLEM is a generative mixture of latent memory framework based on a dynamic mixture-of-experts (MoE)<n>We train the framework on continual-learning sequences spanning math, science, and code domains.<n>After the full continual-learning sequence, our method improves the average accuracy by 10.40% over the Vanilla pretrained baseline.
  arXiv  Detail & Related papers  (2026-05-21T03:35:10Z)
• EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agents [80.59925936278162]
  We present EvolveMem, a self-evolving memory architecture that exposes its full retrieval configuration as a structured action space optimized by an LLM-powered diagnosis module.<n>This closedloop selfevolution realizes an AutoResearch process: the system autonomously conducts iterative research cycles on its own architecture, replacing manual configuration tuning.
  arXiv  Detail & Related papers  (2026-05-13T17:12:44Z)
• Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration [34.98544095454037]
  We train agents to possess an intrinsic meta-evolution capability to spontaneously learn about unseen environments.<n>At inference time, the agent requires no external rewards or human instructions.<n>This shift to native evolution yields a 20% performance increase on WebVoyager and WebWalker.
  arXiv  Detail & Related papers  (2026-04-20T11:54:20Z)
• Mem$^2$Evolve: Towards Self-Evolving Agents via Co-Evolutionary Capability Expansion and Experience Distillation [42.09919382465098]
  Large language model--powered agents can self-evolve by accumulating experience or by dynamically creating new assets.<n>Existing frameworks typically treat these two evolutionary processes in isolation.<n>We introduce a novel paradigm of co-evolutionary Capability Expansion and Experience Distillation.
  arXiv  Detail & Related papers  (2026-04-13T02:44:54Z)
• SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning [83.98129545309277]
  We propose SkillRL, a framework that bridges the gap between raw experience and policy improvement.<n>Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank.<n> Experimental results on ALF, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-09T03:17:17Z)
• MemEvolve: Meta-Evolution of Agent Memory Systems [66.09735157017558]
  Self-evolving memory systems are unprecedentedly reshaping the evolutionary paradigm of large language model (LLM)-based agents.<n>MemeEvolve is a meta-evolutionary framework that jointly evolves agents' experiential knowledge and their memory architecture.<n> EvolveLab is a unified self-evolving memory that distills twelve representative memory systems into a modular design space.
  arXiv  Detail & Related papers  (2025-12-21T14:26:14Z)
• FLEX: Continuous Agent Evolution via Forward Learning from Experience [54.796367711967754]
  Autonomous agents driven by Large Language Models (LLMs) have revolutionized reasoning and problem-solving but remain static after training.<n>We introduce Forward Learning with EXperience (FLEX), a gradient-free learning paradigm that enables LLM agents to continuously evolve through accumulated experience.
  arXiv  Detail & Related papers  (2025-11-09T16:31:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.