Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction

• URL: http://arxiv.org/abs/2605.12070v2
• Date: Sun, 17 May 2026 06:02:48 GMT
• Title: Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
• Abstract summary: We show that practical pipelines with delayed updates and partial rollouts often lose the required historical training-side logits.<n>This missing-old-logit problem entangles discrepancy repair with staleness correction, and makes clipping and masking thresholds interact undesirably.<n>We propose three exact old-logit acquisition strategies: snapshot-based version tracking, a dedicated old-logit model, and synchronization via partial rollout interruption.
• Score: 27.34307252485658
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Asynchronous reinforcement learning improves rollout throughput for large language model agents by decoupling sample generation from policy optimization, but it also introduces a critical failure mode for PPO-style off-policy correction. In heterogeneous training systems, the total importance ratio should ideally be decomposed into two semantically distinct factors: a \emph{training--inference discrepancy term} that aligns inference-side and training-side distributions at the same behavior-policy version, and a \emph{policy-staleness term} that constrains the update from the historical policy to the current policy. We show that practical asynchronous pipelines with delayed updates and partial rollouts often lose the required historical training-side logits, or old logits. This missing-old-logit problem entangles discrepancy repair with staleness correction, breaks the intended semantics of decoupled correction, and makes clipping and masking thresholds interact undesirably. To address this issue, we study both exact and approximate correction routes. We propose three exact old-logit acquisition strategies: snapshot-based version tracking, a dedicated old-logit model, and synchronization via partial rollout interruption, and compare their system trade-offs. From the perspective of approximate correction, we focus on preserving the benefits of decoupled correction through a more appropriate approximate policy when exact old logits cannot be recovered at low cost, without incurring extra system overhead. Following this analysis, we adopt a revised PPO-EWMA method, which achieves significant gains in both training speed and optimization performance.

Related papers

• Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning [6.667218294299622]
  Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization.<n>Existing methods typically retain or discard tokens based solely on the magnitude of their importance ratios.<n>We propose the Entropy-Scaled Trust Region (ESTR), which scales each token's off-policy deviation by its local entropy.
  arXiv  Detail & Related papers  (2026-07-24T10:55:42Z)
• Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models [61.74572554094633]
  We argue that ratio clipping is structurally ill-suited for flow models.<n>We propose Flow-DPPO, which replaces ratio clipping with a divergence proximal constraint.<n> Experiments show that Flow-DPPO achieves higher rewards with better KL-proximal efficiency.
  arXiv  Detail & Related papers  (2026-06-09T15:59:57Z)
• $\oldsymbol{f}$-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control [10.758424473099055]
  Asynchronous execution is necessary for system efficiency, but structurally deviates from the ideal on-policy objective.<n>We propose f-OPD, a novel framework that adaptively regulates stale-sample influence and constrains policy drift.<n>Our results establish the first recipe for achieving a performance-efficiency trade-off in OPD, paving the way for long-horizon agentic post-training at scale.
  arXiv  Detail & Related papers  (2026-05-18T05:14:18Z)
• Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training [50.86545293331458]
  Reinforcement learning is structurally harder than supervised learning because the policy changes the data distribution it learns from.<n>We present a simple yet effective batch-adaptive objective that replaces fixed clipping with the normalized effective sample size of the policy ratios.
  arXiv  Detail & Related papers  (2026-05-12T16:44:47Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training [54.8494905524997]
  Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes.<n>We propose SENTINEL, a verification mechanism for pipeline parallelism (PP) training without duplication.<n>Experiments demonstrate successful training of up to 4B- parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance.
  arXiv  Detail & Related papers  (2026-03-03T23:51:10Z)
• VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training [18.849117699859622]
  Training stability is a central challenge in reinforcement learning for large language models.<n>We propose Variational sEquence-level Soft Policy Optimization (VESPO)<n> Experiments on mathematical reasoning benchmarks show that VESPO maintains stable training under staleness ratios up to 64x and fully asynchronous execution.
  arXiv  Detail & Related papers  (2026-02-11T09:48:08Z)
• Mind the Gap: Data Rewriting for Stable Off-Policy Supervised Fine-Tuning [33.899779762210976]
  Supervised fine-tuning (SFT) of large language models can be viewed as an off-policy learning problem.<n>Existing methods mitigate this issue with KL penalties or clipping, which passively updates rather than actively reducing the gap.<n>We propose a simple yet effective data rewriting framework that proactively shrinks the policy gap before training.
  arXiv  Detail & Related papers  (2025-09-18T17:02:30Z)
• Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance [52.65461207786633]
  Policy-based Reinforcement Learning from Human Feedback is essential for aligning large language models with human preferences.<n>It requires joint training of an actor and critic with a pretrained, fixed reward model for guidance.<n>We propose textbfDecoupled Value Policy Optimization (DVPO), a lean framework that replaces traditional reward modeling with a pretrained emphglobal value model (GVM)
  arXiv  Detail & Related papers  (2025-02-24T08:11:33Z)
• Gumbel Reranking: Differentiable End-to-End Reranker Optimization [61.16471123356738]
  RAG systems rely on rerankers to identify relevant documents.<n> fine-tuning these models remains challenging due to the scarcity of annotated query-document pairs.<n>We propose Gumbel Reranking, an end-to-end training framework for rerankers aimed at minimizing the training-inference gap.
  arXiv  Detail & Related papers  (2025-02-16T13:23:39Z)
• Averaging log-likelihoods in direct alignment [43.77763433288893]
  We introduce a new averaging operator to be composed with the optimality operator giving the best policy for the underlying RL problem. We empirically study the effect of such averaging, observing a trade-off between the length of generations and their scores.
  arXiv  Detail & Related papers  (2024-06-27T14:07:38Z)
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction [96.90215318875859]
  We show that bootstrapping-based Q-learning algorithms do not necessarily benefit from corrective feedback. We propose a new algorithm, DisCor, which computes an approximation to this optimal distribution and uses it to re-weight the transitions used for training.
  arXiv  Detail & Related papers  (2020-03-16T16:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.