Dynamic Important Example Mining for Reinforcement Finetuning
Abstract Overview
This paper studies data utilization in reinforcement fine-tuning (RFT) for large language and vision-language models, arguing that most prior data-centric methods treat sample importance as fixed even though policy learning is non-stationary. The authors propose Dynamic Important Example Mining (DIEM), which adapts sample usage during training through two steps at each optimization iteration: a gradient-alignment estimator of per-sample importance and a constrained reweighting scheme that preserves overall gradient magnitude. The framework is designed to integrate into existing policy-gradient-based RFT methods such as GRPO, PPO, and related variants without requiring a separate selector model. The paper also provides a theoretical error bound for the importance estimator and positions the method as an adaptive, self-organizing curriculum mechanism.
Novelty
The main novelty is a fully automated dynamic example mining framework for RFT that measures each sample's instantaneous value from its gradient alignment with the batch update direction, rather than relying on static selection or heuristic difficulty scores. A second distinctive element is the constrained batch reweighting formulation that emphasizes high-utility samples while explicitly maintaining update-scale stability.
Results
Across math reasoning benchmarks for LLMs, DIEM consistently improves average performance over GRPO on all four reported Qwen models, with gains ranging from +1.68 to +3.36 average points. On multimodal reasoning, DIEM reaches 61.8 average on Qwen2.5-VL-7B and 67.3 on Qwen2.5-VL-32B, outperforming the reported Vanilla RFT and other static or dynamic data-selection baselines. The method is also reported to add only about 1.28% training time overhead in the speed test.
Key Points
- DIEM estimates sample importance during each RFT step using a gradient-alignment proxy for marginal contribution to policy improvement.
- The reweighting procedure is formulated to maximize aggregate utility under a gradient-norm preservation constraint, aiming to keep optimization stable.
- Experiments on both language and vision-language reasoning benchmarks show stronger performance than GRPO, HVS, LIMR, PCL, and SPEED-RL with minimal added training cost.