BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

• URL: http://arxiv.org/abs/2606.01286v1
• Date: Sun, 31 May 2026 15:12:16 GMT
• Title: BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution
• Abstract summary: BenchEvolver is a solution-centric evolutionary framework that transforms existing coding problems into harder variants.<n>We show that BenchEvolver can convert saturated benchmarks into frontier-level evaluation suites and reusable training signal.
• Score: 59.619780973577434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid progress of frontier large language models has led to widespread benchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide useful training signal. For instance, on LiveCodeBench, frontier models achieve over 99% Pass@1 on easy splits and exceed 90% Pass@1 on average across difficulty levels. Constructing new, challenging datasets typically requires substantial human effort, creating a bottleneck for progress. We introduce BenchEvolver, a solution-centric evolutionary framework that automatically transforms existing coding problems into harder variants. Rather than generating problems from scratch, BenchEvolver evolves reference solutions through structured transformations and derives corresponding statements and tests from the evolved solutions. This design grounds generation in executable semantics, enabling scalable construction of high-quality, diverse, and difficult tasks with verifiable correctness. Applying BenchEvolver to LiveCodeBench and SciCode, we obtain evolved tasks that are substantially harder while maintaining validity, reference correctness, and diversity. We further curate LiveCodeBench-Plus, a 91-problem benchmark combining evolved and difficult original LCB-v6 tasks, where frontier-model Pass@1 ranges from 27.5% to 62.6%, restoring clear discrimination among strong coding models. Importantly, evolved tasks remain challenging even for the model that generates them, enabling self-improvement. We further show that RL on evolved LCB tasks improves held-out coding performance: for gpt-oss-20b, seed+evolved training achieves +8.7 and +8.3 Pass@1 gains on LCB v6 Hard and LCB-Pro Easy, exceeding seed-only gains by 70.7% and 34.8%, respectively. Our results show that BenchEvolver can convert saturated benchmarks into frontier-level evaluation suites and reusable training signal.

Related papers

• Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning [51.17262419591891]
  Test-time scaling through iterative self-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains.<n>We present MetaEvolve, a framework designed to develop these meta-skills via a data synthesis pipeline, evolution-aware reinforcement learning, and inference-time evolutionary search.
  arXiv  Detail & Related papers  (2026-07-24T04:35:29Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
  Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
  arXiv  Detail & Related papers  (2026-04-28T16:55:02Z)
• Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problems [70.94268452442147]
  Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of LLMs.<n>We propose a simple yet effective solution based on task reformulation.<n>We introduce Cog-DRIFT, a framework that constructs reformulated variants and organizes them into an adaptive curriculum.
  arXiv  Detail & Related papers  (2026-04-06T15:38:38Z)
• SAGE: Multi-Agent Self-Evolution for LLM Reasoning [34.689664313467595]
  Reinforcement learning with verifiable rewards improves reasoning in large language models (LLMs)<n>We present SAGE, a closed-loop framework where four agents: Challenger, Planner, Solver, and Critic, co-evolve from a shared LLM backbone using only a small seed set.
  arXiv  Detail & Related papers  (2026-03-16T13:24:20Z)
• TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models [26.385183692191873]
  Large Language Models (LLMs) are changing the coding paradigm, yet synthesizingically sophisticated and robust code still remains a critical challenge.<n>We propose Test-driven and cApability-adaptive cuRriculum reinfOrcement fineTuning (TAROT) to address this need.
  arXiv  Detail & Related papers  (2026-02-17T09:29:18Z)
• EvolVE: Evolutionary Search for LLM-based Verilog Generation and Optimization [0.2796197251957245]
  We present EvolVE, the first framework to analyze multiple evolution strategies on chip design tasks.<n>We also introduce IC-RTL, targeting industry-scale problems derived from the National Integrated Circuit Contest.
  arXiv  Detail & Related papers  (2026-01-26T01:53:54Z)
• ThetaEvolve: Test-time Learning on Open Problems [110.5756538358217]
  We introduce ThetaEvolve, an open-source framework that simplifies and extends AlphaEvolve to efficiently scale both in-context learning and Reinforcement Learning (RL) at test time.<n>We find that ThetaEvolve with RL at test-time consistently outperforms inference-only baselines.
  arXiv  Detail & Related papers  (2025-11-28T18:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.