Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism

• URL: http://arxiv.org/abs/2605.23945v1
• Date: Sun, 03 May 2026 05:53:32 GMT
• Title: Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
• Abstract summary: We propose PAT, an adaptive TP method that reconfigures TP during the generation stage of each RLHF iteration.<n>Online reconfiguration mechanism updates only the states and layouts affected by TP changes.<n> PAT reduces generation latency by up to 34.6% and end-to-end RLHF training latency by up to 27.2% compared to the original VeRL setup.
• Score: 11.019309466529643
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning from Human Feedback (RLHF) has become a key post-training paradigm for improving model quality. However, the synchronous three-stage RLHF pipeline is often bottlenecked by the generation stage, where response-length skew causes the effective batch size to shrink rapidly during decoding, leaving GPUs underutilized while a few long responses remain unfinished. Mainstream frameworks employ a static tensor parallelism (TP) configuration that cannot adapt to changing batch characteristics, leaving substantial performance headroom unexplored. We propose PAT, an adaptive TP method that dynamically reconfigures TP during the generation stage of each RLHF iteration. PAT introduces two key techniques. First, a predictor-guided online reconfiguration method decides both the reconfiguration point and the target TP configuration based on offline profiling, triggering reconfiguration only when the predicted latency benefit outweighs the reconfiguration overhead. Second, a lightweight online reconfiguration mechanism updates only the states and layouts affected by TP changes: it adapts unfinished decoding states through a cost-model-based choice between KV-cache migration and recomputation, performs in-place weight resharding, and reuses cached communication groups. We implement PAT on top of SGLang and integrate it with the VeRL framework. Evaluations on LLaMA3.1-8B and Qwen3-14B using DeepScaleR show that PAT reduces generation latency by up to 34.6% and end-to-end RLHF training iteration latency by up to 27.2% compared to the original VeRL setup.

Related papers

• RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers [8.085475675888045]
  RecurrentGPT is a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times.<n>Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate to modulate the recurrent update.<n>Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality.
  arXiv  Detail & Related papers  (2026-08-15T06:22:00Z)
• Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling [87.16803442525755]
  Reinforcement learning (RL) has become a key component in modern large language models, yet the rollout stage remains the key bottleneck in RL training pipelines.<n>Although Multi-Token Prediction (MTP) offers a natural solution to accelerate rollouts through speculative decoding, many studies have observed that MTP acceptance rates degrade significantly during RL training.<n>We present Bebop, a systematic study of MTP in LLM post-training, and offer practical recipes to integrate MTP into large-scale RL pipelines.
  arXiv  Detail & Related papers  (2026-06-10T17:36:45Z)
• Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency [5.093144724934585]
  PACI is a bubble-free pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization.<n>In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, and achieves fully utilized pipeline throughput.<n>These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains.
  arXiv  Detail & Related papers  (2026-06-05T22:33:57Z)
• Joint Training of Multi-Token Prediction in Reinforcement Learning via Optimal Coefficient Calibration [61.46060073417047]
  Multi-Token Prediction (MTP) has been a widely adopted module in pretraining.<n>We show that the per-step effect of MTP on the RL objective can be decomposed into two terms: a first-order correlation and a second-order penalty.<n>We propose an adaptive scheme that tracks the optimal coefficient online via a log-probability proxy at negligible cost.
  arXiv  Detail & Related papers  (2026-05-27T09:07:06Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding [8.43502622452414]
  We study speculative decoding as an acceleration primitive for RL rollouts.<n>We implement speculative decoding in NeMo-RL with a vLLM backend.<n>We project that combining speculative decoding with asynchronous RL yields up to 2.5x end-to-end training speedup at 235B scale.
  arXiv  Detail & Related papers  (2026-04-29T15:11:48Z)
• SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling [54.276306194000405]
  We propose SortedRL, an online length-aware scheduling strategy for scaling reinforcement learning.<n>SortedRL reorders rollout samples based on output lengths, prioritizing short samples forming groups for early updates.<n>Experiments using LLaMA-3.1-8B and Qwen-2.5-32B on diverse tasks, including logical puzzles, show that SortedRL reduces RL training bubble ratios by over 50%.
  arXiv  Detail & Related papers  (2026-03-24T16:48:31Z)
• Efficient and Stable Reinforcement Learning for Diffusion Language Models [59.75789436018925]
  Reinforcement Learning (RL) is crucial for unlocking the complex reasoning capabilities of Diffusion-based Large Language Models (dLLMs)<n>Applying to dLLMs faces unique challenges in efficiency and stability.<n>We propose Spatio-Temporal Pruning (STP), a framework designed to simultaneously improve the efficiency and stability of RL for dLLMs.
  arXiv  Detail & Related papers  (2026-02-09T17:04:23Z)
• Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL [16.40150726450328]
  In bandwidth-constrained decentralized environments, our approach achieves over 100x (14 GB to 108 MB) communication reduction.<n>We present a systematic empirical study of weight-update sparsity at both step-level and multi-step granularities.<n>We find that update sparsity is consistently high, frequently exceeding 99% across practically relevant settings.
  arXiv  Detail & Related papers  (2026-02-03T18:56:48Z)
• Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter [52.111923076688505]
  Training Large Language Models (LLMs) with strong reasoning capabilities marks a significant milestone, unlocking new frontiers in complex problem-solving.<n>We propose TLT, a system that accelerates reasoning RL training losslessly by integrating adaptive speculative decoding.
  arXiv  Detail & Related papers  (2025-11-20T18:59:25Z)
• Laminar: A Scalable Asynchronous RL Post-Training Framework [20.127034898123508]
  Long-tail skewness in RL trajectory generation causes severe GPU underutilization.<n>Current RL systems rely on global weight synchronization between the actor and all rollouts, which creates a rigid model update schedule.<n>We propose Laminar, a scalable and robust RL post-training system built on a fully decoupled architecture.
  arXiv  Detail & Related papers  (2025-10-14T15:29:14Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• RTP: Rethinking Tensor Parallelism with Memory Deduplication [3.036340414461332]
  Rotated Parallelism (RTP) is an innovative approach that focuses on memory deduplication in distributed training environments. Our empirical evaluations underscore RTP's efficiency, revealing that its memory consumption during distributed system training is remarkably close to the optimal.
  arXiv  Detail & Related papers  (2023-11-02T23:12:42Z)
• RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization [30.927648867624498]
  This paper explores training efficient VGG-style super-resolution (SR) networks with the structural re- parameterization technique. Batch normalization (BN) is important to bring training non-linearity and improve the final performance. In particular, we first train SR networks with mini-batch statistics as usual, and then switch to using population statistics at the later training period.
  arXiv  Detail & Related papers  (2022-05-11T17:55:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.