Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning

• URL: http://arxiv.org/abs/2607.21971v1
• Date: Fri, 24 Jul 2026 04:35:29 GMT
• Title: Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
• Abstract summary: Test-time scaling through iterative self-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains.<n>We present MetaEvolve, a framework designed to develop these meta-skills via a data synthesis pipeline, evolution-aware reinforcement learning, and inference-time evolutionary search.
• Score: 51.17262419591891
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Test-time scaling through iterative self-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains. We hypothesize that the success of such evolution frameworks hinges on meta-skills, such as self-reflection with environment feedback, that enable effective multi-round refinement, yet are largely neglected by traditional post-training. To bridge this gap, we present MetaEvolve, a framework designed to develop these meta-skills via a data synthesis pipeline, evolution-aware reinforcement learning (RL), and inference-time evolutionary search. Concretely, we ground MetaEvolve in coding, where program execution provides natural, continuous reward signals beyond binary correctness. Building on these signals, we synthesize evolution trajectories as training data, each containing a current program, its fitness score (combining correctness and efficiency), and a history of prior attempts, and train the model via RL with verifiable rewards derived from test case execution. By training on large-scale code data, we aim to inspire generalizable domain-agnostic meta-skills that can transfer broadly to open-ended problems where such rich training signals are scarce. Across seven coding benchmarks, MetaEvolve outperforms the strongest baseline by 10.01% absolute on in-distribution tasks and 24.12% on out-of-distribution tasks. On open-ended algorithm optimization problems entirely outside the training domain, it further achieves a 46.9% relative improvement. These results demonstrate that explicitly cultivating self-evolution meta-skills offers a principled path toward more capable and autonomously self-evolving AI.

Related papers

• Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents [67.97365535829098]
  Learning from experience is critical for developing capable, self-improving large language model (LLM) agents.<n>We propose Evo-Harness, which distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation.<n>Our analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly.
  arXiv  Detail & Related papers  (2026-08-15T06:43:56Z)
• Evolutionary Discovery of Developmental Reward Schedules in Deep Reinforcement Learning [0.0]
  We propose an evolutionary framework for discovering developmental reward schedules.<n>Agency, novelty, and reactivity are combined through time-varying weights.<n>Our results position evolutionary optimization as a promising approach for developmental reward schedule discovery.
  arXiv  Detail & Related papers  (2026-06-18T18:47:26Z)
• Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
  Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
  arXiv  Detail & Related papers  (2026-06-05T15:09:52Z)
• Learning to Self-Evolve [37.39989112050656]
  We introduce Learning to Self-Evolve (LSE), a reinforcement learning framework that trains large language models (LLMs) to improve their own contexts at test time.<n>LSE reduces the multi-step evolution problem to a single-step RL objective, where each context edit is rewarded by the improvement in downstream performance.<n>Our results highlight the effectiveness of treating self-evolution as a learnable skill.
  arXiv  Detail & Related papers  (2026-03-19T08:41:24Z)
• SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning [83.98129545309277]
  We propose SkillRL, a framework that bridges the gap between raw experience and policy improvement.<n>Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank.<n> Experimental results on ALF, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-09T03:17:17Z)
• Yunjue Agent Tech Report: A Fully Reproducible, Zero-Start In-Situ Self-Evolving Agent System for Open-Ended Tasks [10.622439192272527]
  Conventional agent systems struggle in open-ended environments where task distributions continuously drift and external supervision is scarce.<n>We propose the In-Situ Self-Evolving paradigm, which treats sequential task interactions as a continuous stream of experience.<n>Within this framework, we develop Yunjue Agent, a system that iteratively synthesizes, optimize, and reuses tools to navigate emerging challenges.
  arXiv  Detail & Related papers  (2026-01-26T07:27:47Z)
• Differentiable Evolutionary Reinforcement Learning [41.96953381133274]
  We propose Differentiable Evolutionary Reinforcement Learning (DERL), a bilevel framework that enables the autonomous discovery of optimal reward signals.<n>DERL is differentiable in its metaoptimization: it treats the inner-loop validation performance as a signal to update the Meta-r via reinforcement learning.<n> Experimental results show that DERL achieves state-of-the-art performance on ALFWorld and ScienceWorld, significantly outperforming methods relying on rewards.
  arXiv  Detail & Related papers  (2025-12-15T14:50:08Z)
• ThetaEvolve: Test-time Learning on Open Problems [110.5756538358217]
  We introduce ThetaEvolve, an open-source framework that simplifies and extends AlphaEvolve to efficiently scale both in-context learning and Reinforcement Learning (RL) at test time.<n>We find that ThetaEvolve with RL at test-time consistently outperforms inference-only baselines.
  arXiv  Detail & Related papers  (2025-11-28T18:58:14Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.