Scaling Reasoning Tokens via RL and Parallel Thinking: Evidence From Competitive Programming

• URL: http://arxiv.org/abs/2604.01302v1
• Date: Wed, 01 Apr 2026 18:05:49 GMT
• Title: Scaling Reasoning Tokens via RL and Parallel Thinking: Evidence From Competitive Programming
• Abstract summary: We study how to scale reasoning token budgets for competitive programming through two complementary approaches.<n>We introduce a multi-round parallel thinking pipeline that distributes the token budget across threads and rounds of generation, verification, and refinement.<n>We train the model end-to-end on this pipeline to match the training objective to the test-time structure.
• Score: 10.478834409140623
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study how to scale reasoning token budgets for competitive programming through two complementary approaches: training-time reinforcement learning (RL) and test-time parallel thinking. During RL training, we observe an approximately log-linear relationship between validation accuracy and the average number of generated reasoning tokens over successive checkpoints, and show two ways to shift this training trajectory: verification RL warmup raises the starting point, while randomized clipping produces a steeper trend in the observed regime. As scaling single-generation reasoning during RL quickly becomes expensive under full attention, we introduce a multi-round parallel thinking pipeline that distributes the token budget across threads and rounds of generation, verification, and refinement. We train the model end-to-end on this pipeline to match the training objective to the test-time structure. Starting from Seed-OSS-36B, the full system with 16 threads and 16 rounds per thread matches the underlying RL model's oracle pass@16 at pass@1 using 7.6 million tokens per problem on average, and surpasses GPT-5-high on 456 hard competitive programming problems from AetherCode.

Related papers

• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch [33.82215097004296]
  We propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch.<n>On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.
  arXiv  Detail & Related papers  (2026-04-09T06:11:46Z)
• SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling [54.276306194000405]
  We propose SortedRL, an online length-aware scheduling strategy for scaling reinforcement learning.<n>SortedRL reorders rollout samples based on output lengths, prioritizing short samples forming groups for early updates.<n>Experiments using LLaMA-3.1-8B and Qwen-2.5-32B on diverse tasks, including logical puzzles, show that SortedRL reduces RL training bubble ratios by over 50%.
  arXiv  Detail & Related papers  (2026-03-24T16:48:31Z)
• $V_1$: Unifying Generation and Self-Verification for Parallel Reasoners [69.66089681814013]
  $V_$ is a framework that unifies generation and verification through efficient pairwise ranking.<n>$V_$-Infer improves Pass@1 by up to $10%$ over pointwise verification.<n>$V_$-PairRL achieves $7$--$9%$ test-time scaling gains over standard RL and pointwise joint training.
  arXiv  Detail & Related papers  (2026-03-04T17:22:16Z)
• Beat the long tail: Distribution-Aware Speculative Decoding for RL Training [75.75462952580796]
  We propose a Distribution Aware Speculative decoding framework that accelerates RL rollouts without altering model outputs.<n>Experiments on math and code reasoning tasks show that DAS reduces rollout time up to 50% while preserving identical training curves.
  arXiv  Detail & Related papers  (2025-11-17T19:02:12Z)
• DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation [5.496363733566038]
  We construct RLVR (i.e., RL prompts) and present training techniques that yield strong performance on competitive-programming code generation.<n>We implement our method on Qwen2.5-32B and evaluate on LeetCode and Codeforces weekly contests to avoid data leakage.<n>The resulting model achieves state-of-the-art performance among models of similar scale and is comparable to leading systems such as DeepSeek v3.1 and Doubao-1.5-Thinking.
  arXiv  Detail & Related papers  (2025-11-09T10:11:28Z)
• Rethinking Thinking Tokens: LLMs as Improvement Operators [80.12087211785949]
  Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which allows them to explore solution strategies with self-checking.<n>This results in higher accuracy, but inflates context length, token/compute cost, and answer latency.<n>We ask: Can current models leverage their metacognition to provide other combinations on this Pareto frontier?<n>We identify an interesting inference family Parallel-Distill-Refine (PDR), which performs the following: (i) generate diverse drafts in parallel; (ii) distill them into a bounded, textual workspace; and (iii) refine conditioned on this workspace
  arXiv  Detail & Related papers  (2025-10-01T17:08:59Z)
• Parallel-R1: Towards Parallel Thinking via Reinforcement Learning [65.68667585027232]
  Parallel thinking is a novel approach for enhancing the reasoning capabilities of large language models.<n>We propose textbfParallel-R1, the first reinforcement learning framework that enables parallel thinking behaviors.<n>Our framework employs a progressive curriculum that explicitly addresses the cold-start problem in training parallel thinking.
  arXiv  Detail & Related papers  (2025-09-09T17:59:35Z)
• History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL [14.506189610798929]
  Reinforcement learning (RL) has emerged as a pivotal methodology for enhancing the reasoning capabilities of large language models (LLMs)<n>We introduce RhymeRL, an LLM RL system designed to accelerate RL training with two key innovations.<n>First, to enhance rollout generation, we present HistoSpec, a speculative decoding inference engine.<n>Second, to tackle rollout bubbles, we introduce HistoPipe, a two-tier scheduling strategy.
  arXiv  Detail & Related papers  (2025-08-26T01:42:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.