VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models

• URL: http://arxiv.org/abs/2509.19803v1
• Date: Wed, 24 Sep 2025 06:38:58 GMT
• Title: VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
• Abstract summary: Existing rollout-based reinforcement learning methods fail to explicitly consider LLMs' learning ability for samples of different difficulty levels.<n>We propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards.
• Score: 7.350120815363245
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines.

Related papers

• Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs [31.936873332676743]
  We find that sample difficulty has a non-monotonic effect on RLVR.<n>We propose difficulty-adaptive strategies for hard-sample utilization.
  arXiv  Detail & Related papers  (2026-05-27T12:25:57Z)
• D$^2$Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning [41.94719825003411]
  We propose D$2$Evo, a Dual Difficulty-aware self-Evolution RL framework.<n>In each iteration, our method mines medium-difficulty anchors and trains the Questioner to generate diverse questions at appropriate difficulty levels.<n>Extensive experiments demonstrate that D$2$Evo outperforms existing methods on mathematical reasoning benchmarks with fewer than 2K real mathematical samples.
  arXiv  Detail & Related papers  (2026-05-16T15:16:00Z)
• Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learning [59.25637976883812]
  We introduce a new perspective inspired by cognitive learning theory and propose a novel approach called EasyRL.<n>EasyRL is to simulate the human cognitive acquisition curve by integrating reliable knowledge transfer from easy labeled data.<n> Experimental results on mathematical and scientific benchmarks demonstrate that EasyRL, using only 10% of easy labeled data, consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-04-19T08:02:31Z)
• Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problems [70.94268452442147]
  Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of LLMs.<n>We propose a simple yet effective solution based on task reformulation.<n>We introduce Cog-DRIFT, a framework that constructs reformulated variants and organizes them into an adaptive curriculum.
  arXiv  Detail & Related papers  (2026-04-06T15:38:38Z)
• Rethinking Easy-to-Hard: Limits of Curriculum Learning in Post-Training for Deductive Reasoning [61.81622271655058]
  We present a systematic empirical study of curriculum-based post-training of large language models.<n>We find no robust advantage in difficulty-based sequencing over standard random sampling in either accuracy or response length.<n>Our study suggests that, in the context of deductive reasoning, the specific ordering of training examples plays a negligible role in achieving compositional generalization.
  arXiv  Detail & Related papers  (2026-03-28T10:43:36Z)
• MiniRec: Data-Efficient Reinforcement Learning for LLM-based Recommendation [50.417769112326546]
  MiniRec is a data selection framework tailored for RL-based large language models (LLMs) recommendation.<n>It evaluates sample learnability using key RL signals -- rewards -- pruning samples that are too easy (too high reward) or too difficult (consistently low reward)
  arXiv  Detail & Related papers  (2026-02-04T07:15:49Z)
• Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning [49.22815446849924]
  Large Language Models (LLMs) often struggle with problems that require multi-step reasoning.<n>For small-scale open-source models, Reinforcement Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely sampled.<n>We propose Supervised Reinforcement Learning (SRL), a framework that reformulates problem solving as generating a sequence of logical "actions"
  arXiv  Detail & Related papers  (2025-10-29T22:05:08Z)
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs [53.749193998004166]
  Curriculum learning plays a crucial role in enhancing the training efficiency of large language models.<n>We propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead.
  arXiv  Detail & Related papers  (2025-10-01T15:41:27Z)
• GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning [15.43938821214447]
  Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a powerful paradigm for facilitating the self-improvement of large language models (LLMs)<n>This paper introduces Guided Hybrid Policy Optimization (GHPO), a novel difficulty-aware reinforcement learning framework.<n>GHPO dynamically calibrates task difficulty by employing adaptive prompt refinement to provide targeted guidance.
  arXiv  Detail & Related papers  (2025-07-14T08:10:00Z)
• Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning [52.32193550674408]
  We aim to improve the reasoning capabilities of language models via reinforcement learning (RL)<n>We propose to schedule tasks from easy to hard (E2H), allowing LLMs to build reasoning skills gradually.<n>E2H Reasoner significantly improves the reasoning ability of small LLMs (1.5B to 3B)
  arXiv  Detail & Related papers  (2025-06-07T02:41:54Z)
• TACO: Think-Answer Consistency for Optimized Long-Chain Reasoning and Efficient Data Learning via Reinforcement Learning in LVLMs [50.820065021136024]
  DeepSeek R1 has significantly advanced complex reasoning for large language models (LLMs)<n>Recent methods have attempted to replicate R1's reasoning capabilities in multimodal settings.<n>We propose TACO, a novel reinforcement learning algorithm for visual reasoning.
  arXiv  Detail & Related papers  (2025-05-27T06:30:48Z)
• Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning [3.364797975300393]
  We present Observe-R1, a novel framework aimed at enhancing the reasoning capabilities of multimodal large language models (MLLMs)<n>We construct the NeuraLadder dataset, which is organized and sampled according to the difficulty and complexity of data samples for RL training.<n>Experiments with the Qwen2.5-VL-3B and Qwen2.5-VL-7B models on 20k samples from the NeuraLadder dataset show that Observe-R1 outperforms a series of larger reasoning models on both reasoning and general benchmarks.
  arXiv  Detail & Related papers  (2025-05-18T14:08:03Z)
• DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training [19.701565022644605]
  We present a principled curriculum learning framework grounded in the notion of distribution-level learnability.<n>Our framework prioritizes distributions with either high average advantage (exploitation) or low sample count (exploration)<n>Our experiments show that our framework significantly improves convergence speed and final performance.
  arXiv  Detail & Related papers  (2025-04-13T20:10:27Z)
• Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search [57.28671084993782]
  Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains.<n>Recent studies have shown that increasing test-time computation enhances LLMs' reasoning capabilities.<n>We propose a two-stage training paradigm: 1) a small-scale format tuning stage to internalize the COAT reasoning format and 2) a large-scale self-improvement stage leveraging reinforcement learning.
  arXiv  Detail & Related papers  (2025-02-04T17:26:58Z)
• Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Models [1.2233495442213964]
  Large Language Models (LLMs) offer a promising alternative to mitigate RL sample inefficiency and potentially replace human trainers.<n>We address this limitation by a calibrated guidance system that uses Monte Carlo Dropout to enhance LLM advice reliability.<n>We also develop a novel RL policy shaping method based on dynamic model average entropy to adjust the LLM's influence on RL policies according to guidance uncertainty.
  arXiv  Detail & Related papers  (2024-11-15T22:00:29Z)
• Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization [165.98557106089777]
  A key challenge is to enhance the capabilities of large language models (LLMs) amid a looming shortage of high-quality training data. Our study starts from an empirical strategy for the light continual training of LLMs using their original pre-training data sets. We then formalize this strategy into a principled framework of Instance-Reweighted Distributionally Robust Optimization.
  arXiv  Detail & Related papers  (2024-02-22T04:10:57Z)
• Online Cascade Learning for Efficient Inference over Streams [9.516197133796437]
  Large Language Models (LLMs) have a natural role in answering complex queries about data streams. We propose online cascade learning, the first approach to address this challenge. We formulate the task of learning cascades online as an imitation-learning problem.
  arXiv  Detail & Related papers  (2024-02-07T01:46:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.