Is Next-Chunk Reasoning RL Really Better than SFT? Revisiting Training Strategies under no-CoT Data

• URL: http://arxiv.org/abs/2608.23256v1
• Date: Mon, 24 Aug 2026 13:47:45 GMT
• Title: Is Next-Chunk Reasoning RL Really Better than SFT? Revisiting Training Strategies under no-CoT Data
• Abstract summary: We show that Mixed SFT achieves a clearly higher post-RLVR performance ceiling than next-chunk reasoning RL.<n>We also show that higher pre-RLVR accuracy does not necessarily translate into higher post-RLVR accuracy.
• Score: 48.92748013043833
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work proposes next-chunk reasoning RL for leveraging no-CoT data---corpora such as worked solutions and textbook derivations that contain reasoning-rich content but lack explicit chain-of-thought annotations. The method trains a model to generate implicit reasoning traces and rewards them by their ability to predict the next chunk of text. While promising, existing evaluations primarily compare against conventional SFT baselines, leaving open whether the gains come from the RL formulation itself or from more effectively exposing the model to no-CoT data. We address this question with a controlled study of next-chunk reasoning RL and a simple but previously overlooked alternative: Mixed SFT, a single supervised fine-tuning stage that jointly trains on no-CoT and long-CoT data. Despite its simplicity, Mixed SFT achieves a clearly higher post-RLVR performance ceiling than next-chunk reasoning RL while requiring over 60 times less training compute. The advantage is consistent across in-domain mathematical reasoning and out-of-domain reasoning tasks. Moreover, we show that higher pre-RLVR accuracy does not necessarily translate into higher post-RLVR accuracy, highlighting the need to evaluate no-CoT training strategies in the context of the full post-training pipeline.

Related papers

• Understanding Reasoning from Pretraining to Post-Training [53.890128750021375]
  We use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline.<n>We find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss.<n>We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text.
  arXiv  Detail & Related papers  (2026-07-17T16:31:58Z)
• Stop Thinking, Start Looking: Efficient Post-Training for Multimodal Document Question Answering via Reasoning-Free Alignment [2.8730465903425877]
  We introduce Perception-RFT, a training framework that applies Group Relative Policy Optimization to multimodal document QA.<n>We find that reasoning-enabled models suppress their reasoning traces during training, converging to direct perception-based policies at the 4B parameter scale.<n>We show that an early SFT$rightarrow$RL transition achieves comparable precision with 65% less training data.
  arXiv  Detail & Related papers  (2026-07-16T07:43:39Z)
• Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning [29.726247369977198]
  We argue that data strategy should be aligned with the distinct roles of SFT and RL.<n>We propose a difficulty-aware SFT-then-RL framework that organizes training data into stage-specific sets.
  arXiv  Detail & Related papers  (2026-06-03T05:25:06Z)
• Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models [52.11240605311707]
  Supervised fine-tuning followed by reinforcement learning has become a standard post-training paradigm for large language models.<n>We propose EKSFT (Entropy-KL Selective Fine-Tuning), which selectively masks tokens that exhibit either high entropy or high KL divergence from a reference model.<n> Empirical evaluations on mathematical reasoning benchmarks demonstrate that EKSFT consistently outperforms standard SFT.
  arXiv  Detail & Related papers  (2026-05-28T03:36:05Z)
• Unified Data Selection for LLM Reasoning [66.0118347918729]
  Training Large Language Models (LLMs) for complex, long-CoT reasoning is often bottlenecked by the need for massive high-quality reasoning data.<n>We propose High-Entropy Sum (HES), a training-free metric that quantifies reasoning quality by summing only the entropy of the top (e.g., 0.5%) highest-entropy tokens in each reasoning sample.
  arXiv  Detail & Related papers  (2026-05-21T12:21:41Z)
• Tailored Primitive Initialization is the Secret Key to Reinforcement Learning [61.29280885291581]
  Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs)<n>We argue that initializing LLMs with diverse, high-quality reasoning primitives is essential for achieving stable and sample-efficient RL training.<n>We propose Tailor, a finetuning pipeline that automatically discovers and curates novel reasoning primitives.
  arXiv  Detail & Related papers  (2025-11-16T03:12:40Z)
• Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners [28.039145840787683]
  Supervised fine-tuning (SFT) offers complementary benefits but typically requires large-scale data and risks overfitting.<n>Recent attempts to combine SFT and RL face three main challenges: data inefficiency, algorithm-specific designs, and catastrophic forgetting.<n>We propose a plug-and-play framework that dynamically integrates SFT into RL by selecting challenging examples for SFT.
  arXiv  Detail & Related papers  (2025-10-06T03:01:14Z)
• Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead [20.446287312285648]
  We study whether high SFT scores translate to improved performance after RL.<n>We find high SFT scores can be biased toward simpler or more homogeneous data and are not reliably predictive of subsequent RL gains or scaled-up post-training effectiveness.<n>We study alternative metrics and identify generalization loss on held-out reasoning examples and Pass@large k performance to provide strong proxies for the RL outcome.
  arXiv  Detail & Related papers  (2025-10-02T02:57:00Z)
• Reinforced Latent Reasoning for LLM-based Recommendation [92.56166822197919]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks.<n>Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data.<n>In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning.
  arXiv  Detail & Related papers  (2025-05-25T11:03:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.