JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training

• URL: http://arxiv.org/abs/2604.23838v1
• Date: Sun, 26 Apr 2026 18:45:31 GMT
• Title: JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training
• Abstract summary: JigsawRL decomposes each pipeline into a Sub-Stage Graph that exposes the intra-stage and inter-worker imbalance hidden by stage-level systems.<n>J JigsawRL achieves up to 1.85x throughput over Verl on synchronous RL, 1.54x over StreamRL and AReaL on asynchronous RL, and supports heterogeneous pipelines with moderate latency trade-off.
• Score: 18.390343908577318
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present JigsawRL, a cost-efficient framework that explores Pipeline Multiplexing as a new dimension of RL parallelism. JigsawRL decomposes each pipeline into a Sub-Stage Graph that exposes the intra-stage and inter-worker imbalance hidden by stage-level systems. On this abstraction, JigsawRL resolves multiplexing interference through dynamic resource allocation, eliminates fragmented utilization by migrating long-tail rollouts across workers, and formulates their coordination as a graph scheduling problem solved with a look-ahead heuristic. On 4-64 H100/A100 GPUs across different agentic RL pipelines and models, JigsawRL achieves up to 1.85x throughput over Verl on synchronous RL, 1.54x over StreamRL and AReaL on asynchronous RL, and supports heterogeneous pipelines with moderate latency trade-off.

Related papers

• Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation [26.08473785297375]
  DigenRL is a framework for diffusion-based generative large language models (LLMs)<n>It supports flexible resource allocation, accommodates heterogeneous GPU, and facilitates efficient task scheduling.<n>Experiments show that DigenRL achieves 1.56-2.10x throughput improvements over state-of-the-art diffusion RL systems.
  arXiv  Detail & Related papers  (2026-06-23T09:59:35Z)
• AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs [57.86040075371121]
  Reinforcement learning (RL) is increasingly used to improve the reasoning, coding, and tool-use capabilities of large language models.<n>Agentic RL remains prohibitively expensive.<n>We propose AstraFlow, a dataflow-oriented RL system that replaces conventional trainer-centered control with principled component abstractions.
  arXiv  Detail & Related papers  (2026-05-15T03:13:35Z)
• SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling [54.276306194000405]
  We propose SortedRL, an online length-aware scheduling strategy for scaling reinforcement learning.<n>SortedRL reorders rollout samples based on output lengths, prioritizing short samples forming groups for early updates.<n>Experiments using LLaMA-3.1-8B and Qwen-2.5-32B on diverse tasks, including logical puzzles, show that SortedRL reduces RL training bubble ratios by over 50%.
  arXiv  Detail & Related papers  (2026-03-24T16:48:31Z)
• RLHFless: Serverless Computing for Efficient RLHF [13.743738615300662]
  Reinforcement Learning from Human Feedback (RLHF) has been widely applied to Large Language Model (LLM) post-training.<n>We present RLHFless, the first scalable training framework for synchronous RLHF, built on serverless computing environments.
  arXiv  Detail & Related papers  (2026-02-26T07:45:37Z)
• $π_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models [76.66547858171452]
  $pi_textRL$ is an open-source framework for training flow-based Vision-Language-Action (VLA) models in parallel simulation.<n>$pi_textRL$ boosts few-shot SFT models $pi_0.5$ from 57.6% to 97.6% and from 77.1% to 98.3%, respectively.<n>In ManiSkill, we train $pi_textRL$ in 320 parallel environments, improving $pi_textRL$ from 41.6% to 85.7% and $pi_0.5
  arXiv  Detail & Related papers  (2025-10-29T18:37:39Z)
• RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs [48.94639777633359]
  We present RLBoost, a systematic solution for cost-efficient RL training that harvests preemptible GPU resources.<n> RLBoost increases training throughput by 1.51x-1.97x while improving cost efficiency by 28%-49% compared to using only on-demand GPU resources.
  arXiv  Detail & Related papers  (2025-10-22T04:19:37Z)
• AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework [76.96794548655292]
  Large language models (LLMs) have sparked growing interest in building generalist agents that can learn through online interactions.<n>Applying reinforcement learning (RL) to train LLM agents in multi-turn, multi-task settings remains challenging due to lack of scalable infrastructure and stable training algorithms.<n>We present the AgentRL framework for scalable multi-turn, multi-task agentic RL training.
  arXiv  Detail & Related papers  (2025-10-05T13:40:01Z)
• MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster [6.589537564035392]
  Reinforcement learning (RL) is a paradigm increasingly used to align large language models.<n>In this article, we introduce MindSpeed RL, an effective and efficient system for large-scale RL training.
  arXiv  Detail & Related papers  (2025-07-25T07:11:49Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs)<n>StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks.<n> Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)
• Reinforcement Learning with Graph Attention for Routing and Wavelength Assignment with Lightpath Reuse [49.1574468325115]
  We investigate reinforcement learning for routing and spectrum assignment on fixed-grid networks with flex-rate transponders.<n>We show that RWA-LR has 6% increased throughput when candidate paths are ordered by number of hops, rather than total length.<n>We train an RL agent for RWA-LR with graph attention networks for the policy and value functions to exploit the graph-structured data.
  arXiv  Detail & Related papers  (2025-02-20T17:10:11Z)
• SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores [13.948640763797776]
  We present a novel abstraction on the dataflows of RL training, which unifies diverse RL training applications into a general framework. We develop a scalable, efficient, and distributed RL system called ReaLly scalableRL, which allows efficient and massively parallelized training. SRL is the first in the academic community to perform RL experiments at a large scale with over 15k CPU cores.
  arXiv  Detail & Related papers  (2023-06-29T05:16:25Z)
• RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem [37.38316954355031]
  We re-examine the challenges posed by distributed reinforcement learning. We show that viewing RL as a dataflow problem leads to highly composable and performant implementations. We propose RLlib Flow, a hybrid actor-dataflow programming model for distributed RL.
  arXiv  Detail & Related papers  (2020-11-25T13:28:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.