Scaling Self-Play with Self-Guidance

• URL: http://arxiv.org/abs/2604.20209v1
• Date: Wed, 22 Apr 2026 05:50:23 GMT
• Title: Scaling Self-Play with Self-Guidance
• Abstract summary: Self-Guided Self-Play (SGS) is a self-play algorithm that guides the Conjecturer away from degeneracy.<n>SGS surpasses the solve rate of our strongest baseline in fewer than 80 rounds of self-play.
• Score: 62.13619253976748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM self-play algorithms are notable in that, in principle, nothing bounds their learning: a Conjecturer model creates problems for a Solver, and both improve together. However, in practice, existing LLM self-play methods do not scale well with large amounts of compute, instead hitting learning plateaus. We argue this is because over long training runs, the Conjecturer learns to hack its reward, collapsing to artificially complex problems that do not help the Solver improve. To overcome this, we introduce Self-Guided Self-Play (SGS), a self-play algorithm in which the language model itself guides the Conjecturer away from degeneracy. In SGS, the model takes on three roles: Solver, Conjecturer, and a Guide that scores synthetic problems by their relevance to unsolved target problems and how clean and natural they are, providing supervision against Conjecturer collapse. Our core hypothesis is that language models can assess whether a subproblem is useful for achieving a goal. We evaluate the scaling properties of SGS by running training for significantly longer than prior works and by fitting scaling laws to cumulative solve rate curves. Applying SGS to formal theorem proving in Lean4, we find that it surpasses the asymptotic solve rate of our strongest RL baseline in fewer than 80 rounds of self-play and enables a 7B parameter model, after 200 rounds of self-play, to solve more problems than a 671B parameter model pass@4.

Related papers

• RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning [63.432969627395686]
  RefineRL is a novel approach designed to unleash the self-refinement capabilities of large language models for competitive programming problems.<n>Skeptical-Agent is an iterative self-refinement agent equipped with local execution tools to validate generated solutions against public test cases of CP problems.<n>A reinforcement learning solution incentivizes LLMs to self-refine with only standard RLVR data.
  arXiv  Detail & Related papers  (2026-04-01T11:54:57Z)
• GASP: Guided Asymmetric Self-Play For Coding LLMs [37.79170066221302]
  Asymmetric self-play has emerged as a promising paradigm for post-training large language models.<n>We propose Guided Asymmetric Self-Play (GASP), where grounding is provided by real-data goalpost questions.<n>We improve pass@20 on LiveCodeBench (LCB) by 2.5% over unguided asymmetric self-play.
  arXiv  Detail & Related papers  (2026-03-16T22:13:19Z)
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability [25.507069397981194]
  We show that it is possible to realize bi-level meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful stepping stones.<n>Our results suggest that the ability to generate useful stepping stones does not require the ability to actually solve the hard problems.
  arXiv  Detail & Related papers  (2026-01-26T18:46:56Z)
• DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution [38.660154251426505]
  We introduce DARC (Decoupled Asymmetric Reasoning Curriculum), a two-stage framework that stabilizes the self-evolution process.<n>We show that DARC is model-agnostic, yielding an average improvement of 10.9 points across nine reasoning benchmarks and three backbone models.
  arXiv  Detail & Related papers  (2026-01-20T09:12:27Z)
• Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning [49.290631188365786]
  Scaf-GRPO is a training framework that intervenes when a model's independent learning has plateaued.<n>It boosts the pass@1 score of the Qwen2.5-Math-7B model by a relative 44.3% over a vanilla GRPO baseline.<n>This result demonstrates our framework provides a robust and effective methodology for unlocking a model's ability to solve problems previously beyond its reach.
  arXiv  Detail & Related papers  (2025-10-22T17:41:30Z)
• Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs [102.48588475875749]
  We introduce Generative Self-Refinement (GSR), a novel parallel test-time scaling framework.<n>GSR generates a set of candidate responses in parallel and then performs self-refinement to synthesize a new superior solution.<n>We show that our method achieves state-of-the-art performance across five mathematical benchmarks.
  arXiv  Detail & Related papers  (2025-08-27T06:51:48Z)
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning [95.28059121743831]
  Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for training large language models (LLMs) on complex reasoning tasks.<n>We introduce a Self-aware Weakness-driven problem Synthesis framework (SwS) that systematically identifies model deficiencies and leverages them for problem augmentation.<n>SwS enables robust generalization byempowering the model to self-identify and address its weaknesses in RL, yielding average performance gains of 10.0% and 7.7% on 7B and 32B models.
  arXiv  Detail & Related papers  (2025-06-10T17:02:00Z)
• Self-Evolved Preference Optimization for Enhancing Mathematical Reasoning in Small Language Models [17.673293240849787]
  We introduce SPHERE, a self-evolving data generation pipeline that enhances reasoning in small language models (SLMs)<n> SPHERE operates in three stages: (i) Self-Generation, where the model autonomously constructs problem-solving steps; (ii) Self-Correction, enabling it to identify and rectify errors; and (iii) Diversity Induction, improving robustness through multiple valid reasoning trajectories.<n>We show that SPHERE-trained models achieve significant gains over their base versions and match/surpass GPT-4o on certain benchmarks.
  arXiv  Detail & Related papers  (2025-03-04T14:43:25Z)
• Small Language Models Need Strong Verifiers to Self-Correct Reasoning [69.94251699982388]
  Self-correction has emerged as a promising solution to boost the reasoning performance of large language models (LLMs) This work explores whether small (= 13B) language models (LMs) have the ability of self-correction on reasoning tasks with minimal inputs from stronger LMs.
  arXiv  Detail & Related papers  (2024-04-26T03:41:28Z)
• Can Language Models Solve Olympiad Programming? [40.54366634332231]
  This paper introduces the USACO benchmark with 307 problems from the USA Computing Olympiad. We construct and test a range of LM inference methods for competitive programming for the first time. We find GPT-4 only achieves a 8.7% pass@1 accuracy with zero-shot chain-of-thought prompting.
  arXiv  Detail & Related papers  (2024-04-16T23:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.