Heddle: A Distributed Orchestration System for Agentic RL Rollout

• URL: http://arxiv.org/abs/2603.28101v1
• Date: Mon, 30 Mar 2026 07:01:32 GMT
• Title: Heddle: A Distributed Orchestration System for Agentic RL Rollout
• Abstract summary: Heddle is a trajectory-centric system to optimize agentic rollout execution.<n>Heddle effectively neutralizes the long-tail bottleneck, achieving up to 2.5$times$ higher end-to-end rollout throughput.
• Score: 26.439654927366163
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Agentic Reinforcement Learning (RL) enables LLMs to solve complex tasks by alternating between a data-collection rollout phase and a policy training phase. During rollout, the agent generates trajectories, i.e., multi-step interactions between LLMs and external tools. Yet, frequent tool calls induce long-tailed trajectory generation that bottlenecks rollouts. This stems from step-centric designs that ignore trajectory context, triggering three system problems for long-tail trajectory generation: queueing delays, interference overhead, and inflated per-token time. We propose Heddle, a trajectory-centric system to optimize the when, where, and how of agentic rollout execution. Heddle integrates three core mechanisms: trajectory-level scheduling using runtime prediction and progressive priority to minimize cumulative queueing; trajectory-aware placement via presorted dynamic programming and opportunistic migration during idle tool call intervals to minimize interference; and trajectory-adaptive resource manager that dynamically tunes model parallelism to accelerate the per-token time of long-tail trajectories while maintaining high throughput for short trajectories. Evaluations across diverse agentic RL workloads demonstrate that Heddle effectively neutralizes the long-tail bottleneck, achieving up to 2.5$\times$ higher end-to-end rollout throughput compared to state-of-the-art baselines.

Related papers

• HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning [51.95194664412812]
  Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks.<n>We propose Hierarchical Planning and Information Folding (HIPIF) for long-horizon LLM agent learning.<n>HIPIF trains the agent end-to-end to organize long-horizon execution around explicit subgoals while folding completed subgoal histories to reduce long-context interference.
  arXiv  Detail & Related papers  (2026-06-09T07:35:14Z)
• Libra: Efficient Resource Management for Agentic RL Post-Training [11.701871372256205]
  Reinforcement learning (RL) has emerged as a standard post-training paradigm for shaping large language models (LLMs) into capable agents.<n>In agentic RL, the rollout stage generates trajectories while invoking tools, producing long-tailed and non-stationary workloads.<n>We present Libra, a resource management system to address both challenges via two core mechanisms.
  arXiv  Detail & Related papers  (2026-06-02T03:09:13Z)
• Joint Optimization of Trajectory Control, Resource Allocation, and Task Offloading for Multi-UAV-Assisted IoV [42.14161392443562]
  This paper investigates a multi-Unmanned Aerial Vehicle (UAV) joint base station-assisted Internet of Vehicles (IoV) task offloading system.<n>To minimize system delay, a novel multi-agent reinforcement learning paradigm is proposed.
  arXiv  Detail & Related papers  (2026-05-06T02:59:18Z)
• SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling [54.276306194000405]
  We propose SortedRL, an online length-aware scheduling strategy for scaling reinforcement learning.<n>SortedRL reorders rollout samples based on output lengths, prioritizing short samples forming groups for early updates.<n>Experiments using LLaMA-3.1-8B and Qwen-2.5-32B on diverse tasks, including logical puzzles, show that SortedRL reduces RL training bubble ratios by over 50%.
  arXiv  Detail & Related papers  (2026-03-24T16:48:31Z)
• MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning [42.779100789823055]
  We propose MAGE, a Multi-scale Autoregressive GEneration-based offline RL method.<n>MAGE incorporates a condition-guided multi-scale autoencoder to learn hierarchical trajectory representations.<n>Experiments show that MAGE successfully integrates multi-scale trajectory modeling with conditional guidance.
  arXiv  Detail & Related papers  (2026-02-27T07:56:33Z)
• Alignment in Time: Peak-Aware Orchestration for Long-Horizon Agentic Systems [2.5424331328233207]
  We introduce APEMO (Affect-aware Peak-End Modulation for Orchestration), a runtime scheduling layer.<n>APEMO detects trajectory instability through behavioral proxies and targets repairs at critical segments, such as peak moments and endings.<n>Our results reframe alignment as a temporal control problem, offering a resilient engineering pathway for the development of long-horizon agentic systems.
  arXiv  Detail & Related papers  (2026-02-20T00:16:07Z)
• DLLM Agent: See Farther, Run Faster [94.74432470237817]
  Diffusion large language models (DLLMs) have emerged as an alternative to autoregressive (AR) decoding with appealing efficiency and modeling properties.<n>We study this in a controlled setting by instantiatingDLLM and AR backbones within the same agent workflow.<n>We find thatDLLM Agents are on average over 30% faster end to end than AR agents, with some cases exceeding 8x speedup.
  arXiv  Detail & Related papers  (2026-02-07T09:01:18Z)
• Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning [6.742598086990326]
  Reinforcement Learning (RL) has become critical for advancing modern Large Language Models (LLMs), yet existing synchronous RL systems face severe performance bottlenecks.<n>We present Seer, a novel online context learning system that addresses these challenges by exploiting previously overlooked similarities in output lengths and generation patterns among requests sharing the same prompt.<n>Seer introduces three key techniques: divided rollout for dynamic load balancing, context-aware scheduling, and adaptive grouped speculative decoding.
  arXiv  Detail & Related papers  (2025-11-18T16:12:21Z)
• Beat the long tail: Distribution-Aware Speculative Decoding for RL Training [75.75462952580796]
  We propose a Distribution Aware Speculative decoding framework that accelerates RL rollouts without altering model outputs.<n>Experiments on math and code reasoning tasks show that DAS reduces rollout time up to 50% while preserving identical training curves.
  arXiv  Detail & Related papers  (2025-11-17T19:02:12Z)
• EARL: Efficient Agentic Reinforcement Learning Systems for Large Language Models [10.372430331898608]
  Reinforcement learning (RL) has become a pivotal component of large language model (LLM) post-training.<n>We present EARL, a scalable system for efficient agentic RL.
  arXiv  Detail & Related papers  (2025-10-07T13:52:51Z)
• ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation [57.399685080574756]
  Existing MLLM-based VLN methods rely on imitation learning (IL) and often use DAgger for post-training.<n>We propose ActiveVLN, a VLN framework that explicitly enables active exploration through multi-turn RL.<n>Experiments show that ActiveVLN achieves the largest performance gains over IL baselines compared to both DAgger-based and prior RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-09-16T03:31:46Z)
• Agentic Reinforced Policy Optimization [66.96989268893932]
  Large-scale reinforcement learning with verifiable rewards (RLVR) has demonstrated its effectiveness in harnessing the potential of large language models (LLMs) for single-turn reasoning tasks.<n>Current RL algorithms inadequately balance the models' intrinsic long-horizon reasoning capabilities and their proficiency in multi-turn tool interactions.<n>We propose Agentic Reinforced Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training multi-turn LLM-based agents.
  arXiv  Detail & Related papers  (2025-07-26T07:53:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.