ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning

• URL: http://arxiv.org/abs/2607.24062v1
• Date: Mon, 27 Jul 2026 07:05:10 GMT
• Title: ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
• Abstract summary: Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference.<n>We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range.<n>ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.
• Score: 8.215647507089594
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.

Related papers

• The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning [65.72688364382331]
  Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse.<n>One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision.<n>We introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates.
  arXiv  Detail & Related papers  (2026-06-28T17:40:02Z)
• AIS: Adaptive Importance Sampling for Quantized RL [21.387834718338496]
  Reinforcement learning for large language models (LLMs) is dominated by the cost of rollout generation.<n>This introduces a rollout-training mismatch that biases the policy gradient and can cause training to collapse outright on reasoning benchmarks.<n>We show that the mismatch is nonstationary and acts as a double-edged sword: early in training it provides an exploration bonus, exposing the gradient to trajectories the trainer would otherwise under-sample.<n>We propose Adaptive Importance Sampling (AIS), a correction framework that adjusts the strength of its intervention on a per-batch basis.
  arXiv  Detail & Related papers  (2026-05-13T03:36:57Z)
• Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow [48.48936574810267]
  We present the first comprehensive study of FP8 RL training.<n>We propose Jet-RL, an FP8 RL training framework that enables robust and stable RL optimization.
  arXiv  Detail & Related papers  (2026-01-20T18:54:31Z)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models [73.10315509190623]
  Recent reinforcement learning techniques have yielded impressive reasoning improvements in language models.<n>It remains unclear whether post-training truly extends a model's reasoning ability beyond what it acquires during pre-training.<n>We develop a fully controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training.
  arXiv  Detail & Related papers  (2025-12-08T18:12:10Z)
• Stabilizing Reinforcement Learning with LLMs: Formulation and Practices [61.361819972410046]
  We show why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE.<n>This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training.
  arXiv  Detail & Related papers  (2025-12-01T07:45:39Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers [33.18639109838351]
  We analyze the training-inference consistency of MoE models and identify a notable discrepancy in routing behaviors.<n>We propose Rollout Replay Routing (R3), a method that records routing distributions from the inference engine and replays them during training.<n>R3 significantly reduces training-inference policy KL divergence and mitigates extreme discrepancies without compromising training speed.
  arXiv  Detail & Related papers  (2025-10-13T13:11:27Z)
• Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales [13.818149654692863]
  Reinforcement learning (RL) training is inherently unstable due to factors such as moving targets and high gradient variance.<n>In this work, we improve the stability of RL training by adapting the reverse cross entropy (RCE) from supervised learning for noisy data to define a symmetric RL loss.
  arXiv  Detail & Related papers  (2024-05-27T19:28:33Z)
• Low-Precision Reinforcement Learning [63.930246183244705]
  Low-precision training has become a popular approach to reduce computation time, memory footprint, and energy consumption in supervised learning. In this paper we consider continuous control with the state-of-the-art SAC agent and demonstrate that a na"ive adaptation of low-precision methods from supervised learning fails.
  arXiv  Detail & Related papers  (2021-02-26T16:16:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.