A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curricula

• URL: http://arxiv.org/abs/2603.24202v1
• Date: Wed, 25 Mar 2026 11:23:26 GMT
• Title: A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curricula
• Abstract summary: We introduce a scalable multi-turn synthetic data generation pipeline in which a teacher model iteratively refines problems based on in-context student performance summaries.<n>Compared to single-turn generation, this multi-turn approach substantially improves the yield of valid synthetic problems.<n>We systematically study how task difficulty, curriculum scheduling, and environment diversity interact during RL training across the Llama3.1-8B Instruct and Qwen3-8B Base model families.
• Score: 39.433615386487126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) has emerged as a powerful paradigm for improving large language models beyond supervised fine-tuning, yet sustaining performance gains at scale remains an open challenge, as data diversity and structure, rather than volume alone, become the limiting factor. We address this by introducing a scalable multi-turn synthetic data generation pipeline in which a teacher model iteratively refines problems based on in-context student performance summaries, producing structured difficulty progressions without any teacher fine-tuning. Compared to single-turn generation, this multi-turn approach substantially improves the yield of valid synthetic problems and naturally produces stepping stones, i.e. easier and harder variants of the same core task, that support curriculum-based training. We systematically study how task difficulty, curriculum scheduling, and environment diversity interact during RL training across the Llama3.1-8B Instruct and Qwen3-8B Base model families, with additional scaling experiments on Qwen2.5-32B. Our results show that synthetic augmentation consistently improves in-domain code and in most cases out-of-domain math performance, and we provide empirical insights into how curriculum design and data diversity jointly shape RL training dynamics.

Related papers

• Evolutionary Task Discovery: Advancing Reasoning Frontiers via Skill Composition and Complexity Scaling [10.652489725330401]
  Evoutionary Task Discovery (EvoTD) is a framework that treats data as a directed search over a dual-axis manifold of Algorithmic Skills and Complexity Attributes.<n>We introduce structured evolutionary operators to navigate this space: a Crossover operator that synthesizes novel skill compositions to enhance diversity, and a Parametric Mutation operator that scales structural constraints.<n>EvoTD delivers substantial reasoning gains that generalize consistently across model architectures, pretraining regimes, and scales.
  arXiv  Detail & Related papers  (2026-05-12T07:26:40Z)
• SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility [71.76390626651254]
  We propose SPARD, a framework that establishes an automated, self-paced curriculum by perceiving learning progress to adjust dynamically multi-objective reward weights and data importance.<n>Extensive experiments across multiple benchmarks demonstrate that SPARD significantly enhances model capabilities across all domains.
  arXiv  Detail & Related papers  (2026-04-09T05:37:22Z)
• SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoning [24.80806018678682]
  Reinforcement learning (RL) offers a principled way to enhance the reasoning capabilities of large language models.<n>In practice, RL progress often slows when task difficulty becomes poorly aligned with model capability.<n>We propose a framework that sustains effective learning signals through adaptive environment design.
  arXiv  Detail & Related papers  (2026-01-08T10:42:04Z)
• Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models [71.9060068259379]
  We propose cascaded domain-wise reinforcement learning to build general-purpose reasoning models.<n>Our 14B model, after RL, outperforms its SFT teacher, DeepSeek-R1-0528, on LiveCodeBench v5/v6 Pro and silver-medal performance in the 2025 International Olympiad in Informatics (IOI)
  arXiv  Detail & Related papers  (2025-12-15T18:02:35Z)
• Decouple to Generalize: Context-First Self-Evolving Learning for Data-Scarce Vision-Language Reasoning [41.523848964102]
  Recent vision-language models (VLMs) achieve remarkable reasoning through reinforcement learning (RL)<n>RL provides a feasible solution for realizing continuous self-evolving large vision-language models (LVLMs) in the era of experience.<n>Existing strategies such as synthetic data and self-rewarding mechanisms suffer from limited distributions and alignment difficulties.<n>We propose DoGe, a dual-decoupling framework that guides models to first learn from context rather than problem solving.
  arXiv  Detail & Related papers  (2025-12-07T13:17:31Z)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning [93.00629872970364]
  Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks.<n>We introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions.<n>We study whether difficult problems -- those yielding no RL signals and mixed-quality reasoning traces -- can still be effectively used for training.
  arXiv  Detail & Related papers  (2025-06-05T07:53:59Z)
• Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency [10.942999793311765]
  We investigate in-context learning (ICL) through a meticulous experimental framework that systematically varies task complexity and model architecture.<n>We evaluate four distinct models: a GPT2-style Transformer, a Transformer with FlashAttention mechanism, a convolutional Hyena-based model, and the Mamba state-space model.
  arXiv  Detail & Related papers  (2025-05-10T00:22:40Z)
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining [74.83412846804977]
  Reinforcement learning (RL)-based fine-tuning has become a crucial step in post-training language models.<n>We present a systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch.
  arXiv  Detail & Related papers  (2025-04-10T17:15:53Z)
• Robust Learning with Progressive Data Expansion Against Spurious Correlation [65.83104529677234]
  We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
  arXiv  Detail & Related papers  (2023-06-08T05:44:06Z)
• Improving GANs with A Dynamic Discriminator [106.54552336711997]
  We argue that a discriminator with an on-the-fly adjustment on its capacity can better accommodate such a time-varying task. A comprehensive empirical study confirms that the proposed training strategy, termed as DynamicD, improves the synthesis performance without incurring any additional cost or training objectives.
  arXiv  Detail & Related papers  (2022-09-20T17:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.