Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks

• URL: http://arxiv.org/abs/2608.10357v1
• Date: Tue, 11 Aug 2026 01:30:56 GMT
• Title: Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
• Abstract summary: Long-horizon tool-using agents must reason over user goals, domain policies, tool calls, simulator state, and delayed verifiable rewards.<n>We present SINKFLEX-RL, a modular training system for RL in dual-control tool-use environments.
• Score: 3.5732010112106507
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon tool-using agents must reason over user goals, domain policies, tool calls, simulator state, and delayed verifiable rewards. Reinforcement learning (RL) is a natural fit for this setting, but multi-turn on-policy rollouts create long contexts, while model-specific attention layers may require custom masks and learned sink normalization. We present SINKFLEX-RL, a modular training system for RL in dual-control tool-use environments. The system combines a Gymnasium-compatible environment wrapper, a VERL-style rollout dataflow, group-relative policy optimization without a separate value model, and a sink-aware FlexAttention path designed to preserve model-specific sink scaling under causal and sliding-window masks. In a preliminary Tau2Bench retail run, validation reward (mean@1) rises from 0.25 early in training to $0.44$ later in the observed training window, while training-score and trajectory-reward proxies also trend upward. In a fixed-configuration memory benchmark, the optimized attention path reduces peak VRAM from 28.06GB to 22.52GB at 4096 tokens, a $19.7\%$ reduction, and runs the measured 8192-token configuration using $25.53$~GB where the eager baseline runs out of memory. These results illustrate the value of integrating environment interfaces, RL dataflow, and attention-kernel design for memory-feasible long-horizon agent training.

Related papers

• LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents [56.902747731541695]
  We present LEGO-RL, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization.<n>We evaluate LEGO-RL by training the sparse MoE model Qwen3.5-35B-A3B with GSPO across three native coding-agent harnesses.
  arXiv  Detail & Related papers  (2026-08-18T05:34:35Z)
• TERRA: A Hierarchical Parallel Training and Memory Orchestration Framework for High-Resolution AI-based Earth Modeling [9.997208121319169]
  Training high-resolution AI-based Earth forecasting models is memory-intensive.<n>We present TERRA, a hierarchical parallel training framework for high-resolution Earth forecasting.
  arXiv  Detail & Related papers  (2026-08-15T12:52:48Z)
• Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale [12.736059967085025]
  Modern Deep Learning Recommendation Models (DLRMs) follow scaling laws with sequence length.<n>"Fat Row" paradigm pre-materializes these sequences into every training example, creating a storage and I/O wall.<n>We present a emphversioned late materialization paradigm that eliminates this redundancy by storing UIH once in a normalized, immutable tier.
  arXiv  Detail & Related papers  (2026-04-27T06:41:39Z)
• SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling [54.276306194000405]
  We propose SortedRL, an online length-aware scheduling strategy for scaling reinforcement learning.<n>SortedRL reorders rollout samples based on output lengths, prioritizing short samples forming groups for early updates.<n>Experiments using LLaMA-3.1-8B and Qwen-2.5-32B on diverse tasks, including logical puzzles, show that SortedRL reduces RL training bubble ratios by over 50%.
  arXiv  Detail & Related papers  (2026-03-24T16:48:31Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter [52.111923076688505]
  Training Large Language Models (LLMs) with strong reasoning capabilities marks a significant milestone, unlocking new frontiers in complex problem-solving.<n>We propose TLT, a system that accelerates reasoning RL training losslessly by integrating adaptive speculative decoding.
  arXiv  Detail & Related papers  (2025-11-20T18:59:25Z)
• UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning [78.86567400365392]
  We present Semi-online Reinforcement Learning, a novel paradigm that simulates online RL on offline trajectories.<n>To capture long-term training signals, Semi-online RL introduces discounted future returns into the reward computation.<n>Experiments show that ours Semi-online RL achieves SOTA performance among 7B models across four dynamic benchmarks.
  arXiv  Detail & Related papers  (2025-09-15T03:24:08Z)
• Reducing GPU Memory Fragmentation via Spatio-Temporal Planning for Efficient Large-Scale Model Training [9.775731832789116]
  We introduce STWeaver, a GPU memory allator for deep learning frameworks that reduces fragmentation by exploiting temporal regularity in memory allocation behaviors.<n>Built as a plug PyTorch, STWeaver reduces fragmentation ratio on average by 79.2% (up to 100%) across both dense and sparse models, with negligible overhead.
  arXiv  Detail & Related papers  (2025-07-22T06:39:07Z)
• RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch [23.104546205134103]
  Training deep reinforcement learning (DRL) models usually requires high costs. compressing DRL models possesses immense potential for training acceleration and model deployment. We propose a novel sparse DRL training framework, "the textbfRigged textbfReinforcement textbfLearning textbfLottery" (RLx2)
  arXiv  Detail & Related papers  (2022-05-30T12:18:43Z)
• Online Convolutional Re-parameterization [51.97831675242173]
  We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
  arXiv  Detail & Related papers  (2022-04-02T09:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.