D$^2$Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning

• URL: http://arxiv.org/abs/2605.17037v1
• Date: Sat, 16 May 2026 15:16:00 GMT
• Title: D$^2$Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning
• Abstract summary: We propose D$2$Evo, a Dual Difficulty-aware self-Evolution RL framework.<n>In each iteration, our method mines medium-difficulty anchors and trains the Questioner to generate diverse questions at appropriate difficulty levels.<n>Extensive experiments demonstrate that D$2$Evo outperforms existing methods on mathematical reasoning benchmarks with fewer than 2K real mathematical samples.
• Score: 41.94719825003411
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) has demonstrated potential for enhancing reasoning in large language models (LLMs). However, effective RL training, which requires medium-difficulty training samples, faces two fundamental challenges: Effective Data Scarcity and Dynamic Difficulty Shifts, where medium-difficulty samples are scarce and become trivial as models improve. Existing methods mitigate this scarcity to some extent by generating training samples. However, these approaches suffer from anchor-free generation, ignoring co-evolution, and difficulty mismatch. To address these issues, we propose D$^2$Evo, a Dual Difficulty-aware self-Evolution RL framework. In each iteration, our method mines medium-difficulty anchors based on the current Solver's capability, trains the Questioner to generate diverse questions at appropriate difficulty levels, and jointly optimizes both components to enable progressive reasoning gains. Extensive experiments demonstrate that D$^2$Evo outperforms existing methods on mathematical reasoning benchmarks with fewer than 2K real mathematical samples, and exhibits strong generalization on general reasoning benchmarks.

Related papers

• Better, Faster: Harnessing Self-Improvement in Large Reasoning Models [88.9107786925265]
  We propose HSIR, which effectively Harnesses Self-Improvement in large Reasoning models via two simple-yet-effective approaches.<n>Specifically, HSIR introduces a verify-then-exit sampling strategy to mitigate data imbalance.<n>HSIR also designs an Intrinsic Diversity score to quantify overthinking and filter out the undesired solutions.
  arXiv  Detail & Related papers  (2026-05-24T10:54:46Z)
• Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problems [70.94268452442147]
  Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of LLMs.<n>We propose a simple yet effective solution based on task reformulation.<n>We introduce Cog-DRIFT, a framework that constructs reformulated variants and organizes them into an adaptive curriculum.
  arXiv  Detail & Related papers  (2026-04-06T15:38:38Z)
• LLMs Encode How Difficult Problems Are [4.990590622073335]
  We investigate whether large language models encode problem difficulty in a way that aligns with human judgment.<n>We train linear probes across layers and token positions on 60 models, evaluating on mathematical and coding subsets of Easy2HardBench.
  arXiv  Detail & Related papers  (2025-10-20T22:48:23Z)
• DARO: Difficulty-Aware Reweighting Policy Optimization [18.07946696398167]
  Group Relative Policy Optimization ( GRPO) has emerged as the de facto approach for Reinforcement Learning with Verifiable Rewards (RLVR)<n>We provide a unified view, demonstrating that their reliance on static or overly simplistic weighting schemes tied to sample difficulty prevents adaptation to a model's evolving capabilities.<n>We introduce bfbfDifficulty-Aware Reweighting Policy Optimization (DARO), a method that dynamically adjusts the loss contribution of each difficulty group based on the model's learning state.
  arXiv  Detail & Related papers  (2025-10-10T04:57:15Z)
• VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models [7.350120815363245]
  Existing rollout-based reinforcement learning methods fail to explicitly consider LLMs' learning ability for samples of different difficulty levels.<n>We propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards.
  arXiv  Detail & Related papers  (2025-09-24T06:38:58Z)
• Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding [59.60915947702282]
  Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs)<n>Existing RLVR methods often suffer from exploration inefficiency due to mismatches between the training data's difficulty and the model's capability.<n>We propose SEELE, a novel supervision-aided RLVR framework that dynamically adjusts problem difficulty to stay within the high-efficiency region.
  arXiv  Detail & Related papers  (2025-09-08T17:36:21Z)
• Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective [27.94738910330893]
  Reinforcement learning exhibits potential in enhancing the reasoning abilities of large language models.<n>Existing methods attempt to improve efficiency by scheduling problems based on problem difficulties.<n>This paper introduces $textbfC$ompetence-$textbfD$ifficulty, which enables accurate and stable estimation of problem difficulties.
  arXiv  Detail & Related papers  (2025-05-23T09:15:26Z)
• Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoning [69.64809103333839]
  We investigate how explicitly modeling problem's difficulty prior information shapes the effectiveness of reinforcement learning based fine-tuning for multimodal reasoning.<n>Our approach demonstrates significant performances across various multi-modal mathematical reasoning benchmarks with only 2K+0.6K two-stage training data.
  arXiv  Detail & Related papers  (2025-05-19T15:43:10Z)
• DAST: Difficulty-Aware Self-Training on Large Language Models [68.30467836807362]
  Large Language Models (LLM) self-training methods always under-sample on challenging queries.<n>This work proposes a difficulty-aware self-training framework that focuses on improving the quantity and quality of self-generated responses.
  arXiv  Detail & Related papers  (2025-03-12T03:36:45Z)
• Dynamic Contrastive Distillation for Image-Text Retrieval [90.05345397400144]
  We present a novel plug-in dynamic contrastive distillation (DCD) framework to compress image-text retrieval models. We successfully apply our proposed DCD strategy to two state-of-the-art vision-language pretrained models, i.e. ViLT and METER. Experiments on MS-COCO and Flickr30K benchmarks show the effectiveness and efficiency of our DCD framework.
  arXiv  Detail & Related papers  (2022-07-04T14:08:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.