Learning while Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies

• URL: http://arxiv.org/abs/2605.00416v1
• Date: Fri, 01 May 2026 05:20:26 GMT
• Title: Learning while Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies
• Abstract summary: Generalist robot policies increasingly benefit from large-scale pretraining, but offline data alone is insufficient for robust real-world deployment.<n>We present Learning While Deploying (LWD), a fleet-scale offline-to-online reinforcement learning framework for continual post-training of generalist Vision-Language-Action (VLA) policies.
• Score: 23.266003019334438
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generalist robot policies increasingly benefit from large-scale pretraining, but offline data alone is insufficient for robust real-world deployment. Deployed robots encounter distribution shifts, long-tail failures, task variations, and human correction opportunities that fixed demonstration datasets cannot fully capture. We present Learning While Deploying (LWD), a fleet-scale offline-to-online reinforcement learning framework for continual post-training of generalist Vision-Language-Action (VLA) policies. Starting from a pretrained VLA policy, LWD closes the loop between deployment, shared physical experience, policy improvement, and redeployment by using autonomous rollouts and human interventions collected across a robot fleet. To stabilize learning from heterogeneous, sparse-reward fleet data, LWD combines Distributional Implicit Value Learning (DIVL) for robust value estimation with Q-learning via Adjoint Matching (QAM) for policy extraction in flow-based VLA action generators. We validate LWD on a fleet of 16 dual-arm robots across eight real-world manipulation tasks, including semantic grocery restocking and 3--5 minute long-horizon tasks. A single generalist policy improves as fleet experience accumulates, reaching an average success rate of 95%, with the largest gains on long-horizon tasks.

Related papers

• HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy [22.534022126501373]
  Generalist robot policies have demonstrated strong generalization across robotic manipulation tasks.<n>Recent methods improve Visual-Language-Action (VLA) policies through online reinforcement learning on real robots.<n>World models offer a promising alternative by enabling policy optimization with imagined rollouts.<n>We propose HaWMPO, a closed-loop reinforcement learning pipeline for VLA policy post-training with world models.
  arXiv  Detail & Related papers  (2026-09-09T09:33:25Z)
• Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models [120.24139942108405]
  Scaling robot data is crucial for building generalist Vision-Language-Action (VLA) models.<n>Under a fixed robot-data budget, continued pre-training must turn limited trajectories into transferable visual-action knowledge.<n>We propose VLAct, a VLA-oriented VLM backbone trained on broad, heterogeneous, multi-embodiment robot data.
  arXiv  Detail & Related papers  (2026-08-27T17:59:40Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models [84.73890225707264]
  We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies.<n>Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data.<n>We release an open-source robustness with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.
  arXiv  Detail & Related papers  (2026-05-25T06:31:03Z)
• Jump-Start Reinforcement Learning with Vision-Language-Action Regularization [1.2599533416395767]
  Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation.<n>Current limitations hinder their direct use in fast and precise manipulation.<n>We propose Vision-Language-Action Jump-Starting (VLAJS) to improve exploration and learning efficiency.
  arXiv  Detail & Related papers  (2026-04-15T11:17:54Z)
• RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks [28.827331437876452]
  We present RoboClaw, an agentic robotics framework that unifies data collection, policy learning, and task execution under a single VLM-driven controller.<n>At the policy level, RoboClaw introduces Entangled Action Pairs (EAP), which couple forward manipulation behaviors with inverse recovery actions to form self-resetting loops for autonomous data collection.<n>During deployment, the same agent performs high-level reasoning and dynamically orchestrates learned policy primitives to accomplish long-horizon tasks.
  arXiv  Detail & Related papers  (2026-03-12T05:22:59Z)
• On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning [61.38273866396522]
  Vision-Language-Action models have emerged as a powerful paradigm for general-purpose robot learning.<n>Current methods remain unsuitable for challenging simulated- or physical-world deployments.<n>We introduce a Test-Time Reinforcement Learning for VLAs framework that enables on-the-fly policy adaptation during inference.
  arXiv  Detail & Related papers  (2026-01-11T01:51:30Z)
• SOP: A Scalable Online Post-Training System for Vision-Language-Action Models [15.86316960521611]
  Vision-language-action (VLA) models achieve strong generalization through large-scale pre-training, but real-world deployment requires expert-level task proficiency in addition to broad generality.<n>Existing post-training approaches for VLA models are typically offline, single-robot, or task-specific.<n>We introduce a Scalable Online Post-training system that enables online, distributed, multi-task post-training of generalist VLA models directly in the physical world.
  arXiv  Detail & Related papers  (2026-01-06T14:25:11Z)
• Robotic Control via Embodied Chain-of-Thought Reasoning [86.6680905262442]
  Key limitation of learned robot control policies is their inability to generalize outside their training data.<n>Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models can substantially improve their robustness and generalization ability.<n>We introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLAs, in which we train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features before predicting the robot action.
  arXiv  Detail & Related papers  (2024-07-11T17:31:01Z)
• Robot Fleet Learning via Policy Merging [58.5086287737653]
  We propose FLEET-MERGE to efficiently merge policies in the fleet setting. We show that FLEET-MERGE consolidates the behavior of policies trained on 50 tasks in the Meta-World environment. We introduce a novel robotic tool-use benchmark, FLEET-TOOLS, for fleet policy learning in compositional and contact-rich robot manipulation tasks.
  arXiv  Detail & Related papers  (2023-10-02T17:23:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.