Tree-of-Experience: A Structured Experience-Management Solution for Self-Evolving Agents under Low-Repetition and Implicit-Reward Environments

• URL: http://arxiv.org/abs/2606.06960v1
• Date: Fri, 05 Jun 2026 06:39:16 GMT
• Title: Tree-of-Experience: A Structured Experience-Management Solution for Self-Evolving Agents under Low-Repetition and Implicit-Reward Environments
• Abstract summary: We study a more challenging setting: low-repetition tasks with implicit rewards, where past experience is difficult to reuse and feedback is delayed, noisy, and outcome-level.<n>We propose Tree-of-Experience (ToE), a structured experience-management method that organizes, retrieves, validates, and updates agent experience.
• Score: 7.400600301289333
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Experience-based self-evolution is crucial for LLM agents, but existing benchmarks often assume explicit goals, stable task patterns, and clear feedback. We study a more challenging setting: low-repetition tasks with implicit rewards, where past experience is difficult to reuse and feedback is delayed, noisy, and outcome-level. We introduce \textsc{FinEvolveBench}, a temporally controlled benchmark for financial sentiment prediction that links daily news-driven predictions to future excess returns. We further propose Tree-of-Experience (ToE), a structured experience-management method that organizes, retrieves, validates, and updates agent experience. Experiments show that general-purpose experience mechanisms do not consistently outperform no-experience baselines, while ToE achieves stronger overall performance. These results highlight the importance of structured experience management for self-evolving agents in implicit-reward environments.

Related papers

• S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement? [66.17176838165994]
  Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience.<n>It remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions.<n>We introduce textbfStextsuperscript3Gym, an interactive benchmark for evaluating LLM self-improvement.
  arXiv  Detail & Related papers  (2026-08-31T17:05:41Z)
• Tree-of-Experience: Hierarchical Experience Management for Self-Evolving Agents [7.400600301289333]
  We propose textbfTree-textbfof-textbfExperience (ToE), a structured experience-management framework.<n>ToE organizes the experience into a shared tree of analytical perspectives and reasoning paths.<n>Experiments show ToE substantially improves both problem-solving performance and efficiency.
  arXiv  Detail & Related papers  (2026-08-10T02:49:38Z)
• PATH-Bench: Path-Dependent Evaluation of Lifelong Agents [27.837334107109786]
  PATH-Bench is a benchmark for path-dependent evaluation of lifelong agents.<n>We propose Selective Experience Use (SEU), an agent harness that regulates how path-accumulated experience influences each new task.
  arXiv  Detail & Related papers  (2026-08-02T11:00:04Z)
• Rethinking Experience Utilization in Self-Evolving Language Model Agents [51.10420305280499]
  Self-evolving agents improve by accumulating and reusing experience from past interactions.<n>This paper studies experience utilization as a critical design dimension of self-evolving agents.
  arXiv  Detail & Related papers  (2026-05-08T02:48:49Z)
• Internalizing Agency from Reflective Experience [20.650609947690196]
  LEAFE is a framework that internalizes recovery agency from reflective experience.<n>It consistently improves Pass@1 over the base model and achieves higher Pass@k than outcome-driven baselines.
  arXiv  Detail & Related papers  (2026-03-17T17:50:47Z)
• Large Language Model Agents Are Not Always Faithful Self-Evolvers [84.08646612111092]
  Self-evolving large language model (LLM) agents continually improve by accumulating and reusing past experience.<n>We present the first systematic investigation of experience faithfulness, the causal dependence of an agent's decisions on the experience it is given.
  arXiv  Detail & Related papers  (2026-01-30T01:05:15Z)
• Agent Learning via Early Experience [93.83579011718858]
  A long-term goal of language agents is to learn and improve through their own experience, ultimately outperforming humans in complex, real-world tasks.<n>Most current agents rely on supervised fine-tuning on expert data, which is challenging to scale and generalizes poorly.<n>We study two strategies of using such data: (1) Implicit world modeling, which uses collected states to ground the policy in environment dynamics; and (2) Self-reflection, where the agent learns from its suboptimal actions to improve reasoning and decision-making.
  arXiv  Detail & Related papers  (2025-10-09T17:59:17Z)
• ExGRPO: Learning to Reason from Experience [82.83309610498446]
  Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models.<n>Standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability.<n>In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value.
  arXiv  Detail & Related papers  (2025-10-02T17:31:30Z)
• Iterative Experience Refinement of Software-Developing Agents [81.09737243969758]
  Large language models (LLMs) can leverage past experiences to reduce errors and enhance efficiency. This paper introduces the Iterative Experience Refinement framework, enabling LLM agents to refine experiences iteratively during task execution.
  arXiv  Detail & Related papers  (2024-05-07T11:33:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.