Learning to Reason with Curriculum I: Provable Benefits of Autocurriculum

• URL: http://arxiv.org/abs/2603.18325v1
• Date: Wed, 18 Mar 2026 22:17:22 GMT
• Title: Learning to Reason with Curriculum I: Provable Benefits of Autocurriculum
• Abstract summary: We show that autocurriculum, where the model uses its own performance to decide which problems to focus training on, provably improves upon standard training recipes for both supervised fine-tuning and reinforcement learning.<n>For SFT, we show that autocurriculum requires exponentially fewer reasoning demonstrations than non-adaptive fine-tuning, by focusing teacher supervision on prompts where the current model struggles.<n>For RL fine-tuning, autocurriculum decouples the computational cost from the quality of the reference model, reducing the latter to a burn-in cost that is nearly independent of the target accuracy.
• Score: 44.56791874493398
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chain-of-thought reasoning, where language models expend additional computation by producing thinking tokens prior to final responses, has driven significant advances in model capabilities. However, training these reasoning models is extremely costly in terms of both data and compute, as it involves collecting long traces of reasoning behavior from humans or synthetic generators and further post-training the model via reinforcement learning. Are these costs fundamental, or can they be reduced through better algorithmic design? We show that autocurriculum, where the model uses its own performance to decide which problems to focus training on, provably improves upon standard training recipes for both supervised fine-tuning (SFT) and reinforcement learning (RL). For SFT, we show that autocurriculum requires exponentially fewer reasoning demonstrations than non-adaptive fine-tuning, by focusing teacher supervision on prompts where the current model struggles. For RL fine-tuning, autocurriculum decouples the computational cost from the quality of the reference model, reducing the latter to a burn-in cost that is nearly independent of the target accuracy. These improvements arise purely from adaptive data selection, drawing on classical techniques from boosting and learning from counterexamples, and requiring no assumption on the distribution or difficulty of prompts.

Related papers

• EpiCaR: Knowing What You Don't Know Matters for Better Reasoning in LLMs [9.412828452977553]
  Existing approaches reinforce successful reasoning paths, incurring a substantial calibration cost.<n>This failure has been characterized as a form of model collapse in alignment.<n>We proposeEpiCaR as a training objective that jointly optimize reasoning performance and calibration.
  arXiv  Detail & Related papers  (2026-01-11T06:21:13Z)
• Entropy-Based Adaptive Weighting for Self-Training [15.089334734753677]
  We propose Entropy-Based Adaptive Weighting for Self-Training (EAST)<n>EAST is an adaptive weighting strategy designed to prioritize uncertain data during self-training.<n>We evaluate our approach on GSM8K and MATH benchmarks.
  arXiv  Detail & Related papers  (2025-03-31T10:04:35Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• Training Language Models to Reason Efficiently [12.689316164465472]
  We use reinforcement learning to train large reasoning models to reason efficiently.<n>Our method incentivizes models to minimize unnecessary computational overhead while maintaining accuracy.<n> Experiments on two open-weight large reasoning models demonstrate significant reductions in inference cost while preserving most of the accuracy.
  arXiv  Detail & Related papers  (2025-02-06T19:18:16Z)
• Self-Improvement in Language Models: The Sharpening Mechanism [70.9248553790022]
  We offer a new perspective on the capabilities of self-improvement through a lens we refer to as sharpening.<n>Motivated by the observation that language models are often better at verifying response quality than they are at generating correct responses, we formalize self-improvement as using the model itself as a verifier during post-training.<n>We analyze two natural families of self-improvement algorithms based on SFT and RLHF.
  arXiv  Detail & Related papers  (2024-12-02T20:24:17Z)
• Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision [120.40788744292739]
  We propose a two-player paradigm that separates the roles of reasoning and critique models. We first propose AutoMathCritique, an automated and scalable framework for collecting critique data. We demonstrate that the critique models consistently improve the actor's performance on difficult queries at test-time.
  arXiv  Detail & Related papers  (2024-11-25T17:11:54Z)
• Training Language Models to Self-Correct via Reinforcement Learning [98.35197671595343]
  Self-correction has been found to be largely ineffective in modern large language models (LLMs) We develop a multi-turn online reinforcement learning approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. We find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on MATH and HumanEval.
  arXiv  Detail & Related papers  (2024-09-19T17:16:21Z)
• Fast-ELECTRA for Efficient Pre-training [83.29484808667532]
  ELECTRA pre-trains language models by detecting tokens in a sequence that have been replaced by an auxiliary model. We propose Fast-ELECTRA, which leverages an existing language model as the auxiliary model. Our approach rivals the performance of state-of-the-art ELECTRA-style pre-training methods, while significantly eliminating the computation and memory cost brought by the joint training of the auxiliary model.
  arXiv  Detail & Related papers  (2023-10-11T09:55:46Z)
• Cost-Effective Retraining of Machine Learning Models [2.9461360639852914]
  It is important to retrain a machine learning (ML) model in order to maintain its performance as the data changes over time. This creates a trade-off between retraining too frequently, which leads to unnecessary computing costs, and not retraining often enough. We propose ML systems that make automated and cost-effective decisions about when to retrain an ML model.
  arXiv  Detail & Related papers  (2023-10-06T13:02:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.