AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.18464v1
• Date: Thu, 19 Mar 2026 03:50:45 GMT
• Title: AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models
• Abstract summary: We propose AcceRL, a fully asynchronous and decoupled RL framework.<n>AcceRL is the first to integrate a plug-and-play, trainable world model into a distributed asynchronous RL pipeline.<n> Experiments on the LIBERO benchmark demonstrate that AcceRL achieves state-of-the-art (SOTA) performance.
• Score: 21.204182133307047
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) for large-scale Vision-Language-Action (VLA) models faces significant challenges in computational efficiency and data acquisition. We propose AcceRL, a fully asynchronous and decoupled RL framework designed to eliminate synchronization barriers by physically isolating training, inference, and rollouts. Crucially, AcceRL is the first to integrate a plug-and-play, trainable world model into a distributed asynchronous RL pipeline to generate virtual experiences. Experiments on the LIBERO benchmark demonstrate that AcceRL achieves state-of-the-art (SOTA) performance. Systematically, it exhibits super-linear scaling in throughput and highly efficient hardware utilization. Algorithmically, the world-model-augmented variant delivers unprecedented sample efficiency and robust training stability in complex control tasks.

Related papers

• D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models [17.175523380943606]
  Embodied AI has enabled Vision-Language-Action (VLA) models to excel in multimodal perception and task execution.<n>Applying Reinforcement Learning to these massive models in large-scale distributed environments faces severe systemic bottlenecks.<n>We propose D-VLA, a high-concurrency, low-latency distributed RL framework for large-scale embodied foundation models.
  arXiv  Detail & Related papers  (2026-05-13T09:54:31Z)
• RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure [49.88201789074532]
  Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning.<n>We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure.
  arXiv  Detail & Related papers  (2025-12-27T11:14:23Z)
• AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs [24.96730768606278]
  We present AReaL-Hex, a heterogeneous-aware asynchronous RL training system.<n>It effectively schedules how to execute rollout generation and policy model training over heterogeneous GPUs.<n>It delivers up to 1.50x higher training throughput and 1.46x reduction in training cost.
  arXiv  Detail & Related papers  (2025-11-02T04:17:30Z)
• Laminar: A Scalable Asynchronous RL Post-Training Framework [20.127034898123508]
  Long-tail skewness in RL trajectory generation causes severe GPU underutilization.<n>Current RL systems rely on global weight synchronization between the actor and all rollouts, which creates a rigid model update schedule.<n>We propose Laminar, a scalable and robust RL post-training system built on a fully decoupled architecture.
  arXiv  Detail & Related papers  (2025-10-14T15:29:14Z)
• DRL: Discriminative Representation Learning with Parallel Adapters for Class Incremental Learning [63.65467569295623]
  We propose the Discriminative Representation Learning (DRL) framework to specifically address these challenges.<n>To conduct incremental learning effectively and yet efficiently, the DRL's network is built upon a PTM.<n>Our DRL consistently outperforms other state-of-the-art methods throughout the entire CIL period.
  arXiv  Detail & Related papers  (2025-10-14T03:19:15Z)
• Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony [78.70328630805041]
  ROLL Flash is a system that extends ROLL with native support for asynchronous RL post-training.<n>We show that ROLL Flash significantly improves resource utilization and scalability over synchronous RL post-training.
  arXiv  Detail & Related papers  (2025-10-13T12:41:27Z)
• DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions [6.723690093335988]
  We propose a diffusion-based world model that generates future state-reward trajectories conditioned on the current state, action, and return-to-go.<n>We show that conservative offline RL algorithms such as TD3BC and IQL benefit significantly from training on these augmented trajectories.
  arXiv  Detail & Related papers  (2025-09-23T20:06:26Z)
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning [81.7764584515496]
  Vision-Language-Action (VLA) models have emerged as a powerful paradigm for robotic manipulation.<n>These models face two fundamental challenges: scarcity and high cost of large-scale human-operated robotic trajectories.<n>We introduce SimpleVLA-RL, an efficient reinforcement learning framework tailored for VLA models.
  arXiv  Detail & Related papers  (2025-09-11T17:59:17Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signals [49.17123504516502]
  CurrentReinforcement Fine-tuning (RFT) paradigms for Large Language Models (LLMs) suffer from inefficiency due to redundant exposure of identical queries under uniform data sampling.<n>We propose a Gradient-driven Angle-Informed Navigated RL framework.<n>By leveraging the model's intrinsic angle concentration signal, GAIN-RL dynamically selects training data in each epoch, ensuring consistently impactful gradient updates.
  arXiv  Detail & Related papers  (2025-06-02T21:40:38Z)
• LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training [32.575669924032276]
  Reinforcement Learning (RL) has become the most effective post-training approach for improving the capabilities of Large Language Models (LLMs)<n>We present LlamaRL, a fully distributed, asynchronous RL framework optimized for efficient training of large-scale LLMs.
  arXiv  Detail & Related papers  (2025-05-29T22:14:15Z)
• Scaling Offline RL via Efficient and Expressive Shortcut Models [13.050231036248338]
  offline reinforcement learning (RL) remains challenging due to the iterative nature of their noise sampling processes.<n>We introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models to scale both training and inference.<n>We demonstrate that SORL achieves strong performance across a range of offline RL tasks and exhibits positive scaling behavior with increased test-time compute.
  arXiv  Detail & Related papers  (2025-05-28T20:59:22Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs)<n>StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks.<n> Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.