SmoothRL: Online Reinforcement Learning During Asynchronous Execution

• URL: http://arxiv.org/abs/2608.29768v1
• Date: Sun, 30 Aug 2026 12:56:09 GMT
• Title: SmoothRL: Online Reinforcement Learning During Asynchronous Execution
• Abstract summary: Deploying robot policies in the physical world requires satisfying two fundamental desiderata: reliability and smooth real-time execution.<n>We present SmoothRL, an online reinforcement learning framework that fine-tunes a pretrained policy within an asynchronous inference loop.<n>We evaluate SmoothRL on real-world robotic tasks requiring high precision, as well as highly dynamic tasks that necessitate asynchronous execution.
• Score: 12.24698147742807
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deploying robot policies in the physical world requires satisfying two fundamental desiderata: reliability and smooth real-time execution. However, deploying state-of-the-art generalist models presents challenges on both fronts. Achieving the precision and robustness required for real-world deployment necessitates sample-efficient online reinforcement learning (RL) to adapt pretrained models. Meanwhile, the increasing scale of robot foundation models has led to higher inference latency. To satisfy real-time constraints under high latency, modern systems adopt asynchronous inference with action chunking, overlapping policy computation with chunk execution to hide latency and enable smooth control. Despite their complementary roles, integrating asynchronous execution with gradient-based online RL remains underexplored. We present SmoothRL, an online RL framework that fine-tunes a pretrained policy within an asynchronous inference loop. SmoothRL follows a value-gradient paradigm, directly updating policy parameters using gradients of the action-value function with respect to policy actions. To enable correct optimization under asynchronous execution, SmoothRL explicitly models the asynchronous inference process during training. Specifically, each generated action chunk is partitioned by frame index into three regions: a committed region, consisting of actions committed by the previous inference cycle; an execution region, containing newly generated actions executed by the robot; and a discarded region, containing actions superseded by the next inference cycle. Gradients are propagated only through the execution region, ensuring policy optimization aligns with the trajectory distribution induced by asynchronous execution. We evaluate SmoothRL on real-world robotic tasks requiring high precision, as well as highly dynamic tasks that necessitate asynchronous execution.

Related papers

• Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency [38.99457938189648]
  We introduce a latency-aware framework, Asynchronous RL with Intermediate Information (ARLI), that enables RL-based improvement of generalist policies under inference delays.<n>Our framework builds on asynchronous inference approaches, which interleave action generation with execution to hide latency, and addresses its incompatibility with RL.<n>We evaluate our approach across simulated and real-world manipulation tasks, and find that it enables effective finetuning under inference delays where standard RL fails entirely.
  arXiv  Detail & Related papers  (2026-08-24T21:19:50Z)
• World Action Models in Real Time: An Empirical Study of Smooth Execution via Asynchronous Deployment [0.0]
  We study strategies that overlap model inference with action execution to enable responsive and smooth control.<n>We compare six strategies, including synchronous execution, pure asynchronous switching, post-hoc action blending, denoising-time blending, inference-time velocity guidance, and prefix-conditioned generation.
  arXiv  Detail & Related papers  (2026-08-03T08:24:36Z)
• Aligning Flow Map Policies with Optimal Q-Guidance [50.514994916864275]
  Flow map policies are designed for fast action generation by learning to take arbitrary-size jumps.<n>We derive FLOW MAP Q-GUIDANCE (FMQ), a principled closed-form learning target that is optimal for adapting offline flow map policies under a critic-guided trust-region constraint.<n>FMQ achieves state-of-the-art performance in offline-to-online RL, outperforming the previous one-step policy MVP by a relative improvement of 21.3% on the average success rate.
  arXiv  Detail & Related papers  (2026-05-12T17:12:29Z)
• LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning [90.86828952599147]
  We present LaST-R1, a novel reinforcement learning framework designed to harness "latent reasoning-before-acting" policies.<n>LaST-R1 achieves a near-perfect 99.9% average success rate on the LIBERO benchmark.<n>In real-world deployments, LaST-R1 yields up to a 22.5% average improvement over SOTA supervised fine-tuning approach.
  arXiv  Detail & Related papers  (2026-04-30T17:59:52Z)
• DiscreteRTC: Discrete Diffusion Policies are Natural Asynchronous Executors [57.944744187489185]
  We propose DiscreteRTC, which replaces external corrections with native unmasking.<n>DiscreteRTC is simpler to implement with 0 lines of code for async inpainting, faster at inference with only 0.7x computation compared with generating actions from scratch, and better at execution with 50% higher success rate in real-world dynamic pick task compared with flow-based RTC.
  arXiv  Detail & Related papers  (2026-04-27T23:04:03Z)
• DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks [54.32016216994156]
  We introduce the Causal Latent World Model (CLWM), which employs DINOv3 features as generative targets to disentangle interaction semantics from visual noise.<n>To overcome memory scaling, CLWM features a Dual-State Test-Time Training (TTT) Memory that guarantees a strict $mathcalO(1)$ footprint for long-horizon tasks.<n>We present EmbodiChain, an online framework that establishes the Efficiency Law by injecting an infinite flow of physics-grounded trajectories during training.
  arXiv  Detail & Related papers  (2026-04-13T03:19:36Z)
• Closed-Loop Action Chunks with Dynamic Corrections for Training-Free Diffusion Policy [52.106797722292896]
  We present DCDP, a Dynamic Closed-Loop Diffusion Policy framework that integrates chunk-based action generation with real-time correction.<n>In dynamic PushT simulations, DCDP improves adaptability by 19% without retraining while requiring only 5% additional computation.
  arXiv  Detail & Related papers  (2026-03-02T15:04:18Z)
• RL-VLA$^3$: Reinforcement Learning VLA Accelerating via Full Asynchronism [42.27384804295299]
  Vision-Language-Action (VLA) models have emerged as a crucial pathway towards general embodied intelligence.<n>This paper proposes and implements a fully-asynchronous policy training framework encompassing the entire pipeline from environment interaction to actor policy updates.<n>On the LIBERO benchmark, the framework achieves throughput improvements of up to 59.25% compared to existing synchronous strategies.
  arXiv  Detail & Related papers  (2026-02-05T15:30:23Z)
• Real-Time Robot Execution with Masked Action Chunking [38.37108371991901]
  Real-time execution is essential for cyber-physical systems such as robots.<n>Asynchronous inference has recently emerged as a system-level paradigm for real-time robot manipulation.<n>We propose REMAC, which learns corrective adjustments on the pretrained policy through masked action chunking.
  arXiv  Detail & Related papers  (2026-01-27T23:48:32Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.