Astar: Learning to Propose Evolution Directions for Self-Evolving Industrial AI Systems

• URL: http://arxiv.org/abs/2608.27287v1
• Date: Thu, 27 Aug 2026 15:56:40 GMT
• Title: Astar: Learning to Propose Evolution Directions for Self-Evolving Industrial AI Systems
• Abstract summary: Astar is a training-based approach that learns a specialized evolution-guiding model from the abundant iteration histories of industrial systems.<n>Astar has been deployed in Alibaba's advertising system for evolution direction proposal.
• Score: 22.046339198460334
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern AI systems advance through continuous iteration: a loop of proposing evolution directions, implementing code, training, and evaluation. While the latter three stages are increasingly automated, the starting point --- proposing effective evolution directions --- remains a critical bottleneck that still relies heavily on senior experts. In this work, we explore whether AI can take over this role. We find that general-purpose LLMs, even the advanced GPT-5.5, offer only generic and misaligned suggestions: the required expertise is accumulated through experience rather than explicitly codified, and thus hard to inject directly. To this end, we propose Astar, a training-based approach that learns a specialized evolution-guiding model from the abundant iteration histories of industrial systems. Realizing this idea, however, raises four challenges: sparse supervision, noisy data, a vast direction space, and prohibitively expensive verification. We address them along two fronts. On the data side, we design a pipeline that turns noisy historical commits into a large, clean evolutionary corpus via pairwise sample expansion and noise filtering. On the model side, we train the model through mid-training, SFT, and RL, guiding evolution direction generation with hierarchical hints and using the reward model in RL as a fast surrogate evaluator. Astar has been deployed in Alibaba's Lazada advertising system for evolution direction proposal. Astar-8B achieves a single-proposal success rate of 0.6786 in real-execution evaluation, far exceeding human experts (0.3229) and the strongest general-purpose LLM (0.3071). More importantly, Astar closes the loop and enables fully automatic iteration: it guided 20 consecutive iterations over two weeks, improving offline Hitrate@200 by 23.6%, while an online A/B test yielded relative lifts of 4.86% in GMV and 1.82% in advertising revenue.

Related papers

• Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning [51.17262419591891]
  Test-time scaling through iterative self-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains.<n>We present MetaEvolve, a framework designed to develop these meta-skills via a data synthesis pipeline, evolution-aware reinforcement learning, and inference-time evolutionary search.
  arXiv  Detail & Related papers  (2026-07-24T04:35:29Z)
• APEX: Adaptive Principle EXtraction A Three-Layer Self-Evolution Framework for Production AI Agents [0.0]
  Self-improvement in AI agents has emerged as a key research frontier.<n>We propose APEX, a three-layer co-evolution framework that simultaneously evolves.<n>Our results demonstrate that multi-dimensional co-evolution substantially outperforms single-axis harness optimisation.
  arXiv  Detail & Related papers  (2026-06-13T15:47:27Z)
• What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents [70.6980022118038]
  Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging.<n>Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback.<n>We introduce SERL, a selective environment-reweighted learning framework.
  arXiv  Detail & Related papers  (2026-05-19T07:00:55Z)
• RoboPhD: Evolving Diverse Complex Agents Under Tight Evaluation Budgets [0.2126018197183873]
  We present the first systematic comparison of three optimization paradigms -- Elo tournament selection, GEPA, and greedy hill-climbing.<n>RoboPhD introduces validation-free evolution: instead of splitting the budget between training and validation, it uses Elo competition on training data to simultaneously evaluate agents and drive evolution.<n>We release RoboPhD as a versatile toolkit under the MIT license with a simple optimize_anything() API for evolving diverse complex agents.
  arXiv  Detail & Related papers  (2026-04-06T01:39:13Z)
• RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback [54.39884046754265]
  RetroAgent is an online RL framework that enables agents to master complex interactive environments.<n>Our experiments show that RetroAgent achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-03-09T16:23:33Z)
• ThetaEvolve: Test-time Learning on Open Problems [110.5756538358217]
  We introduce ThetaEvolve, an open-source framework that simplifies and extends AlphaEvolve to efficiently scale both in-context learning and Reinforcement Learning (RL) at test time.<n>We find that ThetaEvolve with RL at test-time consistently outperforms inference-only baselines.
  arXiv  Detail & Related papers  (2025-11-28T18:58:14Z)
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning [125.96848846966087]
  Training large language models (LLMs) as interactive agents presents unique challenges.<n>While reinforcement learning has enabled progress in static tasks, multi-turn agent RL training remains underexplored.<n>We propose StarPO, a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents.
  arXiv  Detail & Related papers  (2025-04-24T17:57:08Z)
• Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play [52.3079697845254]
  eva is the first method that allows language models to adaptively create training prompts in both offline and online RL post-training.<n>We show eva can create effective RL curricula and is robust across ablations.
  arXiv  Detail & Related papers  (2024-10-31T08:15:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.