WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

• URL: http://arxiv.org/abs/2607.17299v1
• Date: Sun, 19 Jul 2026 15:31:13 GMT
• Title: WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning
• Abstract summary: WAR is a workload-aware rollout system that substantially accelerates synchronous agentic RL training.<n>Under low load, WAR enables model-free speculative decoding with SuffixDecoding.<n>Under high load, WAR shifts the optimization focus to cache-aware scheduling.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-horizon rollout generation has become the dominant systems bottleneck in agentic reinforcement learning (RL). As agents interact with environments over many turns, trajectories rapidly grow to tens of thousands of tokens, making synchronous RL training increasingly constrained by rollout. We propose WAR, a workload-aware rollout system that substantially accelerates synchronous agentic RL by jointly optimizing decoding and scheduling. WAR is built on a key observation: the optimal rollout optimization strategy depends on runtime load: (1) Under low load, WAR enables model-free speculative decoding with SuffixDecoding, which reuses suffix patterns from previously completed trajectories as speculative drafts for future rollouts. Unlike model-based drafters, SuffixDecoding introduces no additional draft model and avoids GPU contention with rollout generation. (2) Under high load, where saturated batched decoding leaves limited room for speculative speedup, WAR shifts the optimization focus to cache-aware scheduling. A global scheduler places requests across rollout replicas based on cache locality, trajectory progress and server load, reducing redundant KV-cache recomputation and mitigating load imbalance. By combining decoding-level suffix reuse with system-level rollout scheduling, WAR delivers robust throughput improvements across workload regimes without changing the underlying RL algorithm. WAR improves long-context agentic rollout throughput by 1.4x under low load and up to 1.6x under high load. These results show that WAR removes a major rollout bottleneck in synchronous agentic RL and provides a practical path toward scalable long-context agent training.

Related papers

• EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts [20.460789503141466]
  We present EfficientRollout, a system-aware self-SD framework designed to address this gap for RL rollouts.<n> EfficientRollout reduces rollout and end-to-end latency by up to 19.6% and 12.7%, respectively, over an accelerated AR rollout baseline.
  arXiv  Detail & Related papers  (2026-06-17T11:51:06Z)
• Libra: Efficient Resource Management for Agentic RL Post-Training [11.701871372256205]
  Reinforcement learning (RL) has emerged as a standard post-training paradigm for shaping large language models (LLMs) into capable agents.<n>In agentic RL, the rollout stage generates trajectories while invoking tools, producing long-tailed and non-stationary workloads.<n>We present Libra, a resource management system to address both challenges via two core mechanisms.
  arXiv  Detail & Related papers  (2026-06-02T03:09:13Z)
• Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding [8.43502622452414]
  We study speculative decoding as an acceleration primitive for RL rollouts.<n>We implement speculative decoding in NeMo-RL with a vLLM backend.<n>We project that combining speculative decoding with asynchronous RL yields up to 2.5x end-to-end training speedup at 235B scale.
  arXiv  Detail & Related papers  (2026-04-29T15:11:48Z)
• Heddle: A Distributed Orchestration System for Agentic RL Rollout [26.439654927366163]
  Heddle is a trajectory-centric system to optimize agentic rollout execution.<n>Heddle effectively neutralizes the long-tail bottleneck, achieving up to 2.5$times$ higher end-to-end rollout throughput.
  arXiv  Detail & Related papers  (2026-03-30T07:01:32Z)
• SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling [54.276306194000405]
  We propose SortedRL, an online length-aware scheduling strategy for scaling reinforcement learning.<n>SortedRL reorders rollout samples based on output lengths, prioritizing short samples forming groups for early updates.<n>Experiments using LLaMA-3.1-8B and Qwen-2.5-32B on diverse tasks, including logical puzzles, show that SortedRL reduces RL training bubble ratios by over 50%.
  arXiv  Detail & Related papers  (2026-03-24T16:48:31Z)
• RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure [49.88201789074532]
  Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning.<n>We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure.
  arXiv  Detail & Related papers  (2025-12-27T11:14:23Z)
• Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter [52.111923076688505]
  Training Large Language Models (LLMs) with strong reasoning capabilities marks a significant milestone, unlocking new frontiers in complex problem-solving.<n>We propose TLT, a system that accelerates reasoning RL training losslessly by integrating adaptive speculative decoding.
  arXiv  Detail & Related papers  (2025-11-20T18:59:25Z)
• Beat the long tail: Distribution-Aware Speculative Decoding for RL Training [75.75462952580796]
  We propose a Distribution Aware Speculative decoding framework that accelerates RL rollouts without altering model outputs.<n>Experiments on math and code reasoning tasks show that DAS reduces rollout time up to 50% while preserving identical training curves.
  arXiv  Detail & Related papers  (2025-11-17T19:02:12Z)
• RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs [48.94639777633359]
  We present RLBoost, a systematic solution for cost-efficient RL training that harvests preemptible GPU resources.<n> RLBoost increases training throughput by 1.51x-1.97x while improving cost efficiency by 28%-49% compared to using only on-demand GPU resources.
  arXiv  Detail & Related papers  (2025-10-22T04:19:37Z)
• Laminar: A Scalable Asynchronous RL Post-Training Framework [20.127034898123508]
  Long-tail skewness in RL trajectory generation causes severe GPU underutilization.<n>Current RL systems rely on global weight synchronization between the actor and all rollouts, which creates a rigid model update schedule.<n>We propose Laminar, a scalable and robust RL post-training system built on a fully decoupled architecture.
  arXiv  Detail & Related papers  (2025-10-14T15:29:14Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs)<n>StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks.<n> Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.