The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
Abstract Overview
This paper argues that a key problem in LLM reinforcement learning is not only training instability but an objective mismatch caused by separate training and inference engines. Even with synchronized parameters, the training policy and deployed inference policy can assign different probabilities to the same trajectories, so improving the training-side objective does not necessarily improve the policy actually used at inference time. To address this, the authors propose Monotonic Inference Policy Improvement (MIPI) as the target principle and instantiate it with Monotonic Inference Policy Update (MIPU), a two-step framework. MIPU first constructs sampler-referenced candidate updates and then selectively accepts synchronized updates using an inference-gap-aware validation signal.
Novelty
The distinctive contribution is a shift from optimizing monotonic improvement of the training policy to targeting monotonic improvement of the inference policy under persistent training-inference mismatch. The proposed MIPU framework combines a sampler-referenced update rule with a post-update acceptance test based on an inference-side gap proxy, which the paper presents as an objective-level remedy rather than only a correction to mismatch symptoms.
Results
Under FP8-quantized rollout, a high-mismatch setting, MIPU achieved the best average benchmark scores among the compared methods and was the only method marked as stable on both Qwen3-4B and Qwen3-1.7B. On Qwen3-4B, it reached a 66.71 average score versus 65.66 for LR-decay and 64.42 for the baseline; on Qwen3-1.7B, it reached 53.97 versus 52.23 and 50.86, respectively. Ablations indicate that Step 1 improves candidate update quality while Step 2 filters unreliable synchronized updates, and a random rollback control did not reproduce the stability benefit.
Key Points
- The paper identifies training-inference mismatch as an objective-level problem: training-side improvement does not guarantee improvement of the deployed inference policy.
- MIPU operationalizes this view with two stages: sampler-referenced policy updating and inference-gap-aware acceptance or rollback after synchronization.
- Experiments on mathematical reasoning benchmarks under FP8 rollout show higher average performance and substantially better training stability than baseline GRPO, MIS, and LR-decay.