Rethinking Experience Utilization in Self-Evolving Language Model Agents

• URL: http://arxiv.org/abs/2605.07164v1
• Date: Fri, 08 May 2026 02:48:49 GMT
• Title: Rethinking Experience Utilization in Self-Evolving Language Model Agents
• Abstract summary: Self-evolving agents improve by accumulating and reusing experience from past interactions.<n>This paper studies experience utilization as a critical design dimension of self-evolving agents.
• Score: 51.10420305280499
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-evolving agents improve by accumulating and reusing experience from past interactions. Existing work has largely focused on how experience is constructed, represented, and updated, while paying less attention to how experience should be used during runtime decision-making. As a result, most agents rely on rigid usage strategies, either injecting experience once at initialization or at every step, without considering whether it is needed for the current decision. This paper studies experience utilization as a critical design dimension of self-evolving agents. We ask whether agents benefit from interweaving experience use with decision-making, so that experience is invoked only when additional guidance is needed. To examine this question, we introduce {ExpWeaver}, a lightweight instantiation that leaves experience construction unchanged and modifies only runtime utilization by exposing experience as an optional resource during reasoning. Across four representative frameworks, seven LLM backbones, and three types of environments, ExpWeaver consistently achieves the best performance among different utilization strategies. Reinforcement learning experiments further show that this behavior can be amplified through training. Usage-pattern, causal ablation, and entropy-based analyses reveal that ExpWeaver enables agents to invoke experience selectively, at beneficial decision points, and under higher reasoning uncertainty. Overall, our findings call for a shift from merely studying \emph{what} experience to store toward understanding \emph{how} and \emph{when} experience should enter decision-making.

Related papers

• Tree-of-Experience: A Structured Experience-Management Solution for Self-Evolving Agents under Low-Repetition and Implicit-Reward Environments [7.400600301289333]
  We study a more challenging setting: low-repetition tasks with implicit rewards, where past experience is difficult to reuse and feedback is delayed, noisy, and outcome-level.<n>We propose Tree-of-Experience (ToE), a structured experience-management method that organizes, retrieves, validates, and updates agent experience.
  arXiv  Detail & Related papers  (2026-06-05T06:39:16Z)
• Rethinking Continual Experience Internalization for Self-Evolving LLM Agents [36.80404778289742]
  We find that principle-level experience is more durable than instance-level experience.<n>Step-wise injection significantly outperforms global injection by aligning experience with intermediate decision states.<n>Off-policy context-distillation on high-quality teacher trajectories provides a substantially more stable training signal than on-policy context-distillation.
  arXiv  Detail & Related papers  (2026-06-03T10:30:09Z)
• MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences [39.79876767762482]
  We introduce MuSEAgent, a multimodal reasoning agent that enhances decision-making by extending the capabilities of research agents to discover and leverage stateful experiences.<n>Experiments demonstrate that MuSEAgent consistently outperforms strong trajectory-level experience retrieval baselines on both fine-grained visual perception and complex multimodal reasoning tasks.
  arXiv  Detail & Related papers  (2026-03-29T18:54:31Z)
• Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization [71.41478888201401]
  We propose textbfDual textbfGuidance textbfOptimization(textbfDGO) to improve training effectiveness.
  arXiv  Detail & Related papers  (2026-03-25T08:52:56Z)
• Retrieval-Augmented LLM Agents: Learning to Learn from Experience [16.248836438253814]
  We study how to train retrieval-augmented LLM agents to leverage retrieved trajectories in-context.<n>We establish a robust supervised fine-tuning (SFT) recipe using LoRA that outperforms several state-of-the-art agent training pipelines.<n>Our results demonstrate that this combined approach significantly improves generalization to unseen tasks.
  arXiv  Detail & Related papers  (2026-03-18T20:45:04Z)
• Large Language Model Agents Are Not Always Faithful Self-Evolvers [84.08646612111092]
  Self-evolving large language model (LLM) agents continually improve by accumulating and reusing past experience.<n>We present the first systematic investigation of experience faithfulness, the causal dependence of an agent's decisions on the experience it is given.
  arXiv  Detail & Related papers  (2026-01-30T01:05:15Z)
• Agent Learning via Early Experience [93.83579011718858]
  A long-term goal of language agents is to learn and improve through their own experience, ultimately outperforming humans in complex, real-world tasks.<n>Most current agents rely on supervised fine-tuning on expert data, which is challenging to scale and generalizes poorly.<n>We study two strategies of using such data: (1) Implicit world modeling, which uses collected states to ground the policy in environment dynamics; and (2) Self-reflection, where the agent learns from its suboptimal actions to improve reasoning and decision-making.
  arXiv  Detail & Related papers  (2025-10-09T17:59:17Z)
• Iterative Experience Refinement of Software-Developing Agents [81.09737243969758]
  Large language models (LLMs) can leverage past experiences to reduce errors and enhance efficiency. This paper introduces the Iterative Experience Refinement framework, enabling LLM agents to refine experiences iteratively during task execution.
  arXiv  Detail & Related papers  (2024-05-07T11:33:49Z)
• ExpeL: LLM Agents Are Experiential Learners [57.13685954854463]
  We introduce the Experiential Learning (ExpeL) agent to allow learning from agent experiences without requiring parametric updates.<n>Our agent autonomously gathers experiences and extracts knowledge using natural language from a collection of training tasks.<n>At inference, the agent recalls its extracted insights and past experiences to make informed decisions.
  arXiv  Detail & Related papers  (2023-08-20T03:03:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.