Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability

• URL: http://arxiv.org/abs/2604.06628v1
• Date: Wed, 08 Apr 2026 03:11:16 GMT
• Title: Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability
• Abstract summary: A prevailing narrative in LLM post-training holds that supervised finetuning (SFT) memorizes while reinforcement learning (RL) generalizes.<n>We revisit this claim for reasoning SFT with long chain-of-thought (CoT) supervision and find that cross-domain generalization is not absent but conditional.
• Score: 64.97610656176981
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A prevailing narrative in LLM post-training holds that supervised finetuning (SFT) memorizes while reinforcement learning (RL) generalizes. We revisit this claim for reasoning SFT with long chain-of-thought (CoT) supervision and find that cross-domain generalization is not absent but conditional, jointly shaped by optimization dynamics, training data, and base-model capability. Some reported failures are under-optimization artifacts: cross-domain performance first degrades before recovering and improving with extended training (a dip-and-recovery pattern), so shorttraining checkpoints can underestimate generalization. Data quality and structure both matter: low-quality solutions broadly hurt generalization,while verified long-CoT traces yield consistent cross-domain gains. Model capability is essential: stronger models internalize transferable procedural patterns (e.g., backtracking) even from a toy arithmetic game, while weaker ones imitate surface verbosity. This generalization is asymmetric, however: reasoning improves while safety degrades, reframing the question from whether reasoning SFT generalizes to under what conditions and at what cost.

Related papers

• TD-Grokking: Learning from Zero-Reward Problems by Training-Time Decomposition [10.882920019894451]
  We propose TD-Grokking, a training-time decomposition framework for zero-reward problems.<n>We show that TD-Grokking outperforms vanilla GRPO as well as all baseline approaches.
  arXiv  Detail & Related papers  (2026-06-03T06:40:04Z)
• REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge [83.2858110368572]
  We propose textbfREAL (underlineREgression-underlineAware Reinforcement underlineLThought), a principled RL framework designed to optimize regression rewards.<n>We show that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods.
  arXiv  Detail & Related papers  (2026-03-17T21:19:08Z)
• Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction [49.03500737694832]
  We introduce textbfReinforcement textbfLearning with textbfTurn textbfRLSTA, a generalizable training approach designed to stabilize multi-turn interaction.<n>Experiments show that RLSTA significantly outperforms standard fine-tuning and abstention-based methods.
  arXiv  Detail & Related papers  (2026-03-05T04:04:59Z)
• How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns [51.02752099869218]
  Large Language Models (LLMs) display strikingly different generalization behaviors.<n>We introduce a novel benchmark that decomposes reasoning into atomic core skills.<n>We show that RL-tuned models maintain more stable behavioral profiles and resist collapse in reasoning skills, whereas SFT models exhibit sharper drift and overfit to surface patterns.
  arXiv  Detail & Related papers  (2025-12-30T08:16:20Z)
• Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models [71.9060068259379]
  We propose cascaded domain-wise reinforcement learning to build general-purpose reasoning models.<n>Our 14B model, after RL, outperforms its SFT teacher, DeepSeek-R1-0528, on LiveCodeBench v5/v6 Pro and silver-medal performance in the 2025 International Olympiad in Informatics (IOI)
  arXiv  Detail & Related papers  (2025-12-15T18:02:35Z)
• VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision [9.028503801131933]
  We introduce textbfVariance-textbfControlled textbfOptimization-based textbfREweighting (VCORE)<n>By adopting an optimization-theoretic perspective, VCORE enables a principled and adaptive allocation of supervision across tokens.<n> Empirical evaluations demonstrate that VCORE consistently outperforms existing token reweighting methods.
  arXiv  Detail & Related papers  (2025-10-31T13:19:24Z)
• Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning [75.31797502976802]
  We evaluate over 20 open-weight reasoning-tuned models across a broad suite of tasks.<n>We find that most models that succeed in math fail to transfer their gains to other domains.<n>Our results suggest a need to rethink standard post-training recipes.
  arXiv  Detail & Related papers  (2025-07-01T05:23:05Z)
• ReFT: Reasoning with Reinforced Fine-Tuning [9.80361828538909]
  We propose a simple yet effective approach called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of learning LLMs for reasoning.<n>ReFT first warmups the model with SFT, and then employs on-line reinforcement learning, specifically the PPO algorithm in this paper.<n>Experiments on GSM8K, MathQA, and SVAMP datasets show that ReFT significantly outperforms SFT.
  arXiv  Detail & Related papers  (2024-01-17T04:43:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.