DataFlex-RL: An Evaluation Platform for RLVR Data Policies

• URL: http://arxiv.org/abs/2609.06107v1
• Date: Sat, 05 Sep 2026 14:04:50 GMT
• Title: DataFlex-RL: An Evaluation Platform for RLVR Data Policies
• Abstract summary: Data policies for reinforcement learning with verifiable rewards determine which rollouts are used, how strongly they are weighted, and which domains contribute to subsequent training batches.<n>We introduce DataFlex-RL, an evaluation platform for comparing these choices under a common GRPO recipe.<n>Our primary experiment evaluates 13 configurations across 12 matched seeds using Qwen2.5-7B-Base and 12 mathematics, logic, and science benchmarks.
• Score: 11.073211317719393
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which domains contribute to subsequent training batches. We introduce DataFlex-RL, an evaluation platform for comparing these choices under a common GRPO recipe. Our primary experiment evaluates 13 configurations across 12 matched seeds using Qwen2.5-7B-Base and 12 mathematics, logic, and science benchmarks. Uniform GRPO improves the domain-balanced average accuracy by 7.76 percentage points over the untrained checkpoint. None of the eight rollout-selection or reweighting methods achieves a paired 95% confidence interval that excludes zero relative to uniform sampling, and none of the three adaptive mixtures outperforms a fixed equal mixture at the same level of precision. A corrected 12-seed extension on Llama-3.1-8B-Base places the additional methods on the same score scale as the original controls, but does not reveal a consistent winner in terms of observed mean performance. We also quantify evaluation sensitivity by rescoring nine Qwen2.5-7B-Instruct runs using a math-heavy six-benchmark summary, consisting of five mathematics benchmarks and GPQA-Diamond but no logic benchmark, and comparing it with the domain-balanced 12-benchmark summary. The resulting rankings are negatively correlated, with a correlation coefficient of -0.33, whereas summaries that retain all 12 benchmarks largely agree. Across the controlled settings studied here, changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.

Related papers

• LoRA Fine-Tuned Models for Control Systems Course Q\&A: A Multidimensional Evaluation of Model Scale and Rank Effects [12.865623591533456]
  Large language models (LLMs) are increasingly used in specialized university courses.<n>Direct general-purpose responses may be inconsistently structured and hard to verify.<n>We built a supervised fine-tuning dataset of 360 system-user-assistant conversations.
  arXiv  Detail & Related papers  (2026-09-12T13:04:04Z)
• CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation [49.0564416580726]
  We distill the compositional judgments of a cross-attentive reranker into an embedding model.<n>We show that CORE-RERANKER-8B achieves an 82.7% total average, outperforming Jina-Reranker by 10.7 points.<n>The improvements transfer to the MCMR benchmark without sacrificing retrieval performance on COCO and Flickr30K.
  arXiv  Detail & Related papers  (2026-09-03T16:50:29Z)
• Small Language Models as Judges for Rubric-Based Reinforcement Learning [57.707881387886516]
  We study whether smaller language models can serve as efficient and reliable rubric-based judges.<n>We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges.<n>Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods.
  arXiv  Detail & Related papers  (2026-08-30T20:00:17Z)
• Recursive Agentic Reasoning [4.7490116928645065]
  Test-time reasoning methods such as iterative refinement, decomposition, and repeated sampling are often evaluated in isolation.<n>We introduce a unified view of these methods as recursion operators over an agent's reasoning trace.<n>We evaluate all three operators against a single-pass chain-of-thought baseline under a shared harness with identical prompts, token budgets, and grading code.
  arXiv  Detail & Related papers  (2026-08-25T01:34:08Z)
• UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation [0.0]
  UpliftBench evaluates 12 uplift estimators under an outer-test-isolated, multi-objective protocol.<n>We show the disagreement is substantially about metrics, not models.
  arXiv  Detail & Related papers  (2026-08-02T00:51:44Z)
• A Controlled Study of Decoding-Time Truthfulness Methods on Instruction-Tuned LLMs [3.4007995136788]
  Layer-contrast decoding, inference-time intervention, and learned logit adapters have demonstrated 10-30 point gains on TruthfulQA.<n>Modern instruction-tuned LLMs already achieve substantially higher baselines.<n> Deliberative prompting methods appear more robust in the evaluated regime.
  arXiv  Detail & Related papers  (2026-06-10T14:48:05Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• AIS: Adaptive Importance Sampling for Quantized RL [21.387834718338496]
  Reinforcement learning for large language models (LLMs) is dominated by the cost of rollout generation.<n>This introduces a rollout-training mismatch that biases the policy gradient and can cause training to collapse outright on reasoning benchmarks.<n>We show that the mismatch is nonstationary and acts as a double-edged sword: early in training it provides an exploration bonus, exposing the gradient to trajectories the trainer would otherwise under-sample.<n>We propose Adaptive Importance Sampling (AIS), a correction framework that adjusts the strength of its intervention on a per-batch basis.
  arXiv  Detail & Related papers  (2026-05-13T03:36:57Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Bit-Identical Medical Deep Learning via Structured Orthogonal Initialization [0.0]
  Deep learning training is non-deterministic: identical code with different random seeds produces models that disagree on aggregate metrics but disagree on individual predictions.<n>We present a framework for verified bit-identical training that eliminates three sources of randomness.
  arXiv  Detail & Related papers  (2026-03-30T05:04:57Z)
• Benchmark^2: Systematic Evaluation of LLM Benchmarks [66.2731798872668]
  We propose Benchmark2, a comprehensive framework comprising three complementary metrics.<n>We conduct experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains.<n>Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction can achieve comparable evaluation performance.
  arXiv  Detail & Related papers  (2026-01-07T14:59:03Z)
• Aioli: A Unified Optimization Framework for Language Model Data Mixing [74.50480703834508]
  We show that no existing method consistently outperforms a simple stratified sampling baseline in terms of average test perplexity.<n>We derive a new online method named Aioli, which directly estimates the mixing law parameters throughout training and uses them to dynamically adjust proportions.
  arXiv  Detail & Related papers  (2024-11-08T17:50:24Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.