CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoning

• URL: http://arxiv.org/abs/2605.24331v1
• Date: Sat, 23 May 2026 01:18:55 GMT
• Title: CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoning
• Abstract summary: Context or prompt-level reweighting has emerged as a central algorithmic lever in Reinforcement Learning with Verified Rewards.<n>We propose a distribution-aware prompt reweighting approach, called CurveRL, based on a quantile coordinate transform.<n>Our study identifies context-distribution control as a principled axis for analyzing and designing prompt-reweighted RLVR algorithms.
• Score: 20.44110921033217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Context or prompt-level reweighting has emerged as a central algorithmic lever in Reinforcement Learning with Verified Rewards (RLVR) for improving the reasoning capability of large language models, yet the principle determining what constitutes an optimal weighting remains poorly understood. We address this gap by formulating prompt reweighting as a functional derivative of a utility functional defined in the pass-rate function space, yielding a unified optimality framework that accommodates existing schemes, including REINFORCE and GRPO. Building on this optimality framework, we propose a distribution-aware prompt reweighting approach, called CurveRL, based on a quantile coordinate transform, in which the weight assigned to each prompt depends not on the absolute value of pass rates but on its rank and density to reflect the distributional structure of the pass rates in the learning dynamics. Extensive experiments across multiple benchmarks demonstrate that our proposed CurveRL consistently outperforms GRPO and other RLVR baselines. Our study identifies context-distribution control as a principled axis for analyzing and designing prompt-reweighted RLVR algorithms. The code is released in https://github.com/zhyzmath/CurveRL.

Related papers

• Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR [53.27792011950384]
  Reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable paradigm for improving the reasoning capabilities of large language models.<n>We propose NudgeRL, a framework for structured and diversity-driven exploration in RLVR.<n>Our approach introduces Strategy Nudging, which conditions each rollout on lightweight, strategy-level contexts.
  arXiv  Detail & Related papers  (2026-05-15T08:22:59Z)
• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards [73.44333771806282]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision.<n>CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors.
  arXiv  Detail & Related papers  (2026-05-14T08:22:21Z)
• Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVR [23.417956258945427]
  We propose a post-training framework that leverages accuracy estimates to prioritize informative question prompts during training.<n>PACED-RL is a promising direction for a more sample efficient distribution-matching training for LLMs.
  arXiv  Detail & Related papers  (2026-02-13T06:04:14Z)
• Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards [69.74686029941881]
  Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose a unified neural scheduling framework that adaptively selects high-value rollouts throughout training.<n>Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods.
  arXiv  Detail & Related papers  (2026-02-09T10:51:58Z)
• CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs [31.371566320424552]
  CoBA-RL is a reinforcement learning algorithm designed to adaptively allocate rollout budgets based on the model's evolving capability.<n>Our approach effectively orchestrates the trade-off between exploration and exploitation, delivering consistent generalization improvements.
  arXiv  Detail & Related papers  (2026-02-03T03:14:36Z)
• Revisiting LLM Reasoning via Information Bottleneck [57.519119962528166]
  Large language models (LLMs) have recently demonstrated remarkable progress in reasoning capabilities through reinforcement learning with verifiable rewards (RLVR)<n>We present a theoretical characterization of LLM reasoning grounded in information bottleneck (IB) principle.<n>We propose IB-aware reasoning optimization (IBRO), a framework that encourages reasoning trajectories to be both informative about the final correct answer and generalizable.
  arXiv  Detail & Related papers  (2025-07-24T13:14:25Z)
• One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF [2.1212179660694104]
  We primarily examine a variety of RL-based and RL-free methods designed to address Reinforcement Learning from Human Feedback (RLHF) and Large Reasoning Models (LRMs)<n>We reinterpret several RL-based and RL-free algorithms through the perspective of neural structured bandit prediction.<n>This leads to a detailed derivation of the standard RLHF objective within a full RL context, demonstrating its equivalence to neural structured bandit prediction.
  arXiv  Detail & Related papers  (2025-03-25T10:23:26Z)
• REBEL: Reinforcement Learning via Regressing Relative Rewards [59.68420022466047]
  We propose REBEL, a minimalist RL algorithm for the era of generative models.<n>In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL.<n>We find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO.
  arXiv  Detail & Related papers  (2024-04-25T17:20:45Z)
• Provable Reward-Agnostic Preference-Based Reinforcement Learning [61.39541986848391]
  Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories. We propose a theoretical reward-agnostic PbRL framework where exploratory trajectories that enable accurate learning of hidden reward functions are acquired.
  arXiv  Detail & Related papers  (2023-05-29T15:00:09Z)
• One-Step Distributional Reinforcement Learning [10.64435582017292]
  We present the simpler one-step distributional reinforcement learning (OS-DistrRL) framework. We show that our approach comes with a unified theory for both policy evaluation and control. We propose two OS-DistrRL algorithms for which we provide an almost sure convergence analysis.
  arXiv  Detail & Related papers  (2023-04-27T06:57:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.