Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration

• URL: http://arxiv.org/abs/2604.18131v1
• Date: Mon, 20 Apr 2026 11:54:20 GMT
• Title: Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration
• Abstract summary: We train agents to possess an intrinsic meta-evolution capability to spontaneously learn about unseen environments.<n>At inference time, the agent requires no external rewards or human instructions.<n>This shift to native evolution yields a 20% performance increase on WebVoyager and WebWalker.
• Score: 34.98544095454037
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most agents today ``self-evolve'' by following rewards and rules defined by humans. However, this process remains fundamentally dependent on external supervision; without human guidance, the evolution stops. In this work, we train agents to possess an intrinsic meta-evolution capability to spontaneously learn about unseen environments prior to task execution. To instill this ability, we design an outcome-based reward mechanism that measures how much an agent's self-generated world knowledge improves its success rate on downstream tasks. This reward signal is used exclusively during the training phase to teach the model how to explore and summarize effectively. At inference time, the agent requires no external rewards or human instructions. It spontaneously performs native self-evolution to adapt to unknown environments using its internal parameters. When applied to Qwen3-30B and Seed-OSS-36B, this shift to native evolution yields a 20% performance increase on WebVoyager and WebWalker. Most strikingly, the generated world knowledge even enables a compact 14B Qwen3 model to outperform the unassisted Gemini-2.5-Flash, establishing a new paradigm for truly evolving agents.

Related papers

• Aspire: Can Models Self-Evolve from Vague Goals? [61.65007006747275]
  We introduce ASPIRE, a benchmark for vague-goal-driven self-evolution.<n>ASPIRE supports both model-weight and agent-harness evolution in a unified interactive environment.<n>Our experiments show that vague goals redirect search effort toward goal interpretation.
  arXiv  Detail & Related papers  (2026-08-31T17:14:59Z)
• Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning [51.17262419591891]
  Test-time scaling through iterative self-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains.<n>We present MetaEvolve, a framework designed to develop these meta-skills via a data synthesis pipeline, evolution-aware reinforcement learning, and inference-time evolutionary search.
  arXiv  Detail & Related papers  (2026-07-24T04:35:29Z)
• OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation [71.57258488733974]
  Memory has become a standard substrate for self-evolving agents, yet retaining experience is not the same as learning how to evolve through it.<n>We introduce OPD-Evolver, a slow-fast co-evolution framework that cultivates such an agent evolver through on-policy self-distillation.
  arXiv  Detail & Related papers  (2026-06-16T07:33:53Z)
• Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
  Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
  arXiv  Detail & Related papers  (2026-06-05T15:09:52Z)
• Guided Self-Evolving LLMs with Minimal Human Supervision [53.111086364268566]
  Unguided self-evolving systems often plateau quickly or degrade as training.<n>R-Few is a guided Self-Play Challenger acquirer framework that incorporates lightweight human oversight through in-context grounding and mixed training.<n>R-Few achieves consistent and iterative improvements across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-12-02T07:06:11Z)
• Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents [58.69865074060139]
  We study the case where an agent's self-evolution deviates in unintended ways, leading to undesirable or even harmful outcomes.<n>Our empirical findings reveal that misevolution is a widespread risk, affecting agents built even on top-tier LLMs.<n>We discuss potential mitigation strategies to inspire further research on building safer and more trustworthy self-evolving agents.
  arXiv  Detail & Related papers  (2025-09-30T14:55:55Z)
• Building Self-Evolving Agents via Experience-Driven Lifelong Learning: A Framework and Benchmark [57.59000694149105]
  We introduce Experience-driven Lifelong Learning (ELL), a framework for building self-evolving agents.<n>ELL is built on four core principles: Experience Exploration, Long-term Memory, Skill Learning and Knowledge Internalization.<n>We also introduce StuLife, a benchmark dataset for ELL that simulates a student's holistic college journey.
  arXiv  Detail & Related papers  (2025-08-26T13:04:28Z)
• Memento No More: Coaching AI Agents to Master Multiple Tasks via Hints Internalization [56.674356045200696]
  We propose a novel method to train AI agents to incorporate knowledge and skills for multiple tasks without the need for cumbersome note systems or prior high-quality demonstration data.<n>Our approach employs an iterative process where the agent collects new experiences, receives corrective feedback from humans in the form of hints, and integrates this feedback into its weights.<n>We demonstrate the efficacy of our approach by implementing it in a Llama-3-based agent that, after only a few rounds of feedback, outperforms advanced models GPT-4o and DeepSeek-V3 in tasksets.
  arXiv  Detail & Related papers  (2025-02-03T17:45:46Z)
• REvolve: Reward Evolution with Large Language Models using Human Feedback [6.4550546442058225]
  Large language models (LLMs) have been used for reward generation from natural language task descriptions.<n>LLMs, guided by human feedback, can be used to formulate reward functions that reflect human implicit knowledge.<n>We introduce REvolve, a truly evolutionary framework that uses LLMs for reward design in reinforcement learning.
  arXiv  Detail & Related papers  (2024-06-03T13:23:27Z)
• Ego-Foresight: Self-supervised Learning of Agent-Aware Representations for Improved RL [26.169030913260084]
  We present Ego-Foresight, a self-supervised method for disentangling agent and environment based on motion and prediction.<n>Our main finding is self-supervised agent-awareness by visuomotor prediction of the agent improves sample-efficiency and performance of the underlying RL algorithm.
  arXiv  Detail & Related papers  (2024-05-27T13:32:43Z)
• Generative Adversarial Neuroevolution for Control Behaviour Imitation [3.04585143845864]
  We propose to explore whether deep neuroevolution can be used for behaviour imitation on popular simulation environments. We introduce a simple co-evolutionary adversarial generation framework, and evaluate its capabilities by evolving standard deep recurrent networks. Across all tasks, we find the final elite actor agents capable of achieving scores as high as those obtained by the pre-trained agents.
  arXiv  Detail & Related papers  (2023-04-03T16:33:22Z)
• Mimicking Evolution with Reinforcement Learning [10.35437633064506]
  We argue that the path to developing artificial human-like-intelligence will pass through mimicking the evolutionary process in a nature-like simulation. This work proposes Evolution via Evolutionary Reward (EvER) that allows learning to single-handedly drive the search for policies with increasingly evolutionary fitness.
  arXiv  Detail & Related papers  (2020-03-31T18:16:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.