Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents

• URL: http://arxiv.org/abs/2605.10663v1
• Date: Mon, 11 May 2026 14:43:56 GMT
• Title: Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents
• Abstract summary: Self-evolving agents aim to overcome the static nature of large language models by distilling reusable experience from past interactions.<n>We propose Evolving-RL, an efficient algorithmic framework that jointly improves the experience extraction and utilization capabilities required for self-evolution.<n> Experiments on ALFWorld and Mind2Web show that Evolving-RL effectively enhances LLMs' ability to extract and reuse experience.
• Score: 31.6974589324286
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Experience-driven self-evolving agents aim to overcome the static nature of large language models by distilling reusable experience from past interactions, thus enabling adaptation to novel tasks at deployment time. This process places substantial demands on the foundation model's capacities for abstraction, generalization, and in-context learning. However, most existing studies focus primarily on system-level design choices, such as how experience is represented and managed, neglecting the inherent capabilities of the underlying model. While some recent works have started to optimize the experience utilization stage via reinforcement learning, they still fail to treat self-evolution as a unified process to be jointly optimized. To this end, we propose Evolving-RL, an efficient algorithmic framework that jointly improves the experience extraction and utilization capabilities required for self-evolution. Specifically, we center the learning process on experience extraction and evaluation, using the two supervisory signals derived from evaluation to optimize the extractor and solver separately and thus enable their coordinated co-evolution. Experiments on ALFWorld and Mind2Web show that Evolving-RL effectively enhances LLMs' ability to extract and reuse experience, leading to strong performance gains on out-of-distribution tasks (up to 98.7% relative improvement over the GRPO baseline on ALFWorld unseen tasks and 35.8% on Mind2Web), and these gains are fully unlocked only through the coordinated co-evolution of experience extraction and utilization. Furthermore, Evolving-RL inherently functions as an experience-augmented RL algorithm. By internalizing reusable experience patterns directly into model parameters, it achieves remarkable performance gains over standard baselines on both seen and unseen tasks, even in the absence of test-time experience accumulation.

Related papers

• Self-Improving Large Language Models via Progressive Experience Evolution [52.03479747358687]
  We propose textbfSPEE (textbfSelf-textbfProgressive textbfExperience textbfEvolution), a unified post-training framework that sequentially performs explicit experience evolution followed by implicit policy optimization.<n> Experiments on five mathematical reasoning benchmarks demonstrate that SPEE consistently outperforms both test-time and training-time self-evolution baselines across three model scales.
  arXiv  Detail & Related papers  (2026-08-03T12:27:32Z)
• MetaEvo: A Meta-Optimization Framework for Experience-Driven Agent Evolution [49.24192816558635]
  Large language models (LLMs) exhibit strong reasoning capabilities, yet most LLM-based agents are unable to improve through task interactions.<n>Existing experience-driven methods often rely on memory or executors without enhancing the model's ability to learn.<n>We propose MetaEvo, a two-stage framework for continual agent evolution.
  arXiv  Detail & Related papers  (2026-05-29T09:31:39Z)
• EvoIR-Agent: Self-Evolving Image Restoration Agentic System via Experience-Driven Learning [39.60899845728349]
  Multimodal Large Language Model (MLLM)-driven image restoration agent demonstrates effectiveness in degradation coupling scenarios.<n>Training-based methods embed intrinsic experience into parameters, achieving high inference efficiency but lacking compatibility with new tools or degradation.<n>We propose EvoIR-Agent, which first systematically formulates the experience components of a training-free image restoration agent.
  arXiv  Detail & Related papers  (2026-05-21T09:14:25Z)
• Experiential Reflective Learning for Self-Improving LLM Agents [1.1074589887824053]
  We introduce Experiential Reflective Learning (ERL), a simple self-improvement framework that enables rapid environment adaptation.<n>ERL reflects on task trajectories and outcomes to generate actionable lessons that transfer across tasks.<n>ERL improves success rate by 7.8% over a ReAct baseline, with large gains in task completion reliability.
  arXiv  Detail & Related papers  (2026-03-25T11:43:22Z)
• Complementary Reinforcement Learning [31.660877399506493]
  Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based agents.<n>Existing approaches suffer from a critical weakness: the experience distilled from history is either stored statically or fail to coevolve with the improving actor.<n>Inspired by complementary learning systems in neuroscience, we present Complementary RL to achieve seamless co-evolution of an experience extractor and a policy actor.
  arXiv  Detail & Related papers  (2026-03-18T11:38:01Z)
• Online Experiential Learning for Language Models [99.90826536842337]
  Online Experiential Learning (OEL) is a framework that enables language models to continuously improve from their own deployment experience.<n>OEL operates in two stages: first, transferable experiential knowledge is extracted and accumulated from interaction trajectories collected on the user side.<n>We evaluate OEL on text-based game environments across multiple model scales and both thinking and non-thinking variants.
  arXiv  Detail & Related papers  (2026-03-17T17:57:49Z)
• RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback [54.39884046754265]
  RetroAgent is an online RL framework that enables agents to master complex interactive environments.<n>Our experiments show that RetroAgent achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-03-09T16:23:33Z)
• ExGRPO: Learning to Reason from Experience [82.83309610498446]
  Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models.<n>Standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability.<n>In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value.
  arXiv  Detail & Related papers  (2025-10-02T17:31:30Z)
• Iterative Experience Refinement of Software-Developing Agents [81.09737243969758]
  Large language models (LLMs) can leverage past experiences to reduce errors and enhance efficiency. This paper introduces the Iterative Experience Refinement framework, enabling LLM agents to refine experiences iteratively during task execution.
  arXiv  Detail & Related papers  (2024-05-07T11:33:49Z)
• Learning to Optimize for Reinforcement Learning [58.01132862590378]
  Reinforcement learning (RL) is essentially different from supervised learning, and in practice, these learneds do not work well even in simple RL tasks. Agent-gradient distribution is non-independent and identically distributed, leading to inefficient meta-training. We show that, although only trained in toy tasks, our learned can generalize unseen complex tasks in Brax.
  arXiv  Detail & Related papers  (2023-02-03T00:11:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.