DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training

• URL: http://arxiv.org/abs/2606.30345v1
• Date: Mon, 29 Jun 2026 14:20:47 GMT
• Title: DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training
• Abstract summary: DRIFT is an online self-evolution policy optimization framework for large language models.<n>It regulates the model's self-improvement process through the joint use of Difficulty Routing and Rhythm Gating.<n> evaluated across five benchmarks and three model scales, DRIFT surpasses the peak performance of both GRPO and SDPO.
• Score: 12.82389963785782
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Enabling large language models to achieve stable self-improvement without external expert supervision remains a central challenge in complex reasoning tasks. Existing self-distillation and reinforcement learning methods lack explicit mechanisms for tracking problem-level learning progress and adapting optimization strategies accordingly. Consequently, training may over-optimize easy problems, receive weak supervision from hard problems, and fail to sufficiently explore borderline cases. To resolve these issues, we propose DRIFT, an online self-evolution policy optimization framework for large language models. DRIFT regulates the model's self-improvement process through the joint use of Difficulty Routing and Rhythm Gating. The former identifies the model's learning state at the problem level and dynamically allocates self-distillation and reinforcement learning signals, while the latter refines policy updates at the token level, concentrating exploration on critical reasoning positions. By further incorporating a success buffer and a two-stage curriculum learning strategy, DRIFT preserves high-quality historical experience while progressively guiding the model from reliable behavior acquisition toward stable policy evolution. Evaluated across five benchmarks and three model scales, DRIFT surpasses the peak performance of both GRPO and SDPO across all evaluated metrics. On the average score over the five benchmarks, DRIFT achieves 79.5$\%$, outperforming GRPO by 9.5$\%$ and SDPO by 7.5$\%$, establishing a new state-of-the-art result. Notably, on ToolUse, DRIFT reaches an accuracy of 79.2$\%$, improving over GRPO by 13.5$\%$ and SDPO by 10.7$\%$, setting a new state-of-the-art and substantially outperforming all concurrent methods.

Related papers

• Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problems [70.94268452442147]
  Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of LLMs.<n>We propose a simple yet effective solution based on task reformulation.<n>We introduce Cog-DRIFT, a framework that constructs reformulated variants and organizes them into an adaptive curriculum.
  arXiv  Detail & Related papers  (2026-04-06T15:38:38Z)
• Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing [79.88256756334327]
  Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision.<n>We propose Sample-Routed Policy Optimization (SRPO)<n>SRPO routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction.
  arXiv  Detail & Related papers  (2026-04-02T17:29:18Z)
• ROAD: Reflective Optimization via Automated Debugging for Zero-Shot Agent Alignment [1.6968020497268546]
  ROAD is a novel framework that treats optimization as a dynamic debug investigation rather than a search.<n>Road is highly sample-efficient, achieving a 5.6 percent increase in success rate and a 3.8 percent increase in search accuracy.<n>These findings suggest that mimicking the human engineering loop of failure analysis and patching offers a viable, data-efficient alternative to resource-intensive training.
  arXiv  Detail & Related papers  (2025-12-30T07:31:34Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail [85.47497935739936]
  Alpamayo-R1 (AR1) is a vision-language-action model that integrates Chain of Causation reasoning with trajectory planning.<n>We show AR1 achieves 12% improvement in planning accuracy on challenging cases compared to a trajectory-only baseline.<n>We plan to release AR1 models and a subset of the CoC in a future update.
  arXiv  Detail & Related papers  (2025-10-30T01:25:34Z)
• Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning [49.290631188365786]
  Scaf-GRPO is a training framework that intervenes when a model's independent learning has plateaued.<n>It boosts the pass@1 score of the Qwen2.5-Math-7B model by a relative 44.3% over a vanilla GRPO baseline.<n>This result demonstrates our framework provides a robust and effective methodology for unlocking a model's ability to solve problems previously beyond its reach.
  arXiv  Detail & Related papers  (2025-10-22T17:41:30Z)
• DARO: Difficulty-Aware Reweighting Policy Optimization [18.07946696398167]
  Group Relative Policy Optimization ( GRPO) has emerged as the de facto approach for Reinforcement Learning with Verifiable Rewards (RLVR)<n>We provide a unified view, demonstrating that their reliance on static or overly simplistic weighting schemes tied to sample difficulty prevents adaptation to a model's evolving capabilities.<n>We introduce bfbfDifficulty-Aware Reweighting Policy Optimization (DARO), a method that dynamically adjusts the loss contribution of each difficulty group based on the model's learning state.
  arXiv  Detail & Related papers  (2025-10-10T04:57:15Z)
• Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning [12.155067662797281]
  Pentest-R1 is a framework designed to optimize reasoning capabilities for penetration testing tasks.<n>It learns directly from environmental feedback to develop robust error self-correction and adaptive strategies.<n>On AutoPenBench, Pentest-R1 achieves a 24.2% success rate, surpassing most state-of-the-art models.
  arXiv  Detail & Related papers  (2025-08-10T15:14:05Z)
• SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents [58.174206358223415]
  Self-Evolving Embodied Agents-R1, or SEEA-R1, is the first reinforcement fine-tuning framework designed for self-evolving embodied agents.<n>We show that SEEA-R1 can support autonomous adaptation and reward-driven self-evolution.
  arXiv  Detail & Related papers  (2025-06-26T18:00:07Z)
• From Imitation to Exploration: End-to-end Autonomous Driving based on World Model [24.578178308010912]
  RAMBLE is an end-to-end world model-based RL method for driving decision-making.<n>It can handle complex and dynamic traffic scenarios.<n>It achieves state-of-the-art performance in route completion rate on the CARLA Leaderboard 1.0 and completes all 38 scenarios on the CARLA Leaderboard 2.0.
  arXiv  Detail & Related papers  (2024-10-03T06:45:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.