SOP: A Scalable Online Post-Training System for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2601.03044v1
• Date: Tue, 06 Jan 2026 14:25:11 GMT
• Title: SOP: A Scalable Online Post-Training System for Vision-Language-Action Models
• Abstract summary: Vision-language-action (VLA) models achieve strong generalization through large-scale pre-training, but real-world deployment requires expert-level task proficiency in addition to broad generality.<n>Existing post-training approaches for VLA models are typically offline, single-robot, or task-specific.<n>We introduce a Scalable Online Post-training system that enables online, distributed, multi-task post-training of generalist VLA models directly in the physical world.
• Score: 15.86316960521611
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language-action (VLA) models achieve strong generalization through large-scale pre-training, but real-world deployment requires expert-level task proficiency in addition to broad generality. Existing post-training approaches for VLA models are typically offline, single-robot, or task-specific, limiting effective on-policy adaptation and scalable learning from real-world interaction. We introduce a Scalable Online Post-training (SOP) system that enables online, distributed, multi-task post-training of generalist VLA models directly in the physical world. SOP tightly couples execution and learning through a closed-loop architecture in which a fleet of robots continuously streams on-policy experience and human intervention signals to a centralized cloud learner, and asynchronously receives updated policies. This design supports prompt on-policy correction, scales experience collection through parallel deployment, and preserves generality during adaptation. SOP is agnostic to the choice of post-training algorithm; we instantiate it with both interactive imitation learning (HG-DAgger) and reinforcement learning (RECAP). Across a range of real-world manipulation tasks including cloth folding, box assembly, and grocery restocking, we show that SOP substantially improves the performance of large pretrained VLA models while maintaining a single shared policy across tasks. Effective post-training can be achieved within hours of real-world interaction, and performance scales near-linearly with the number of robots in the fleet. These results suggest that tightly coupling online learning with fleet-scale deployment is instrumental to enabling efficient, reliable, and scalable post-training of generalist robot policies in the physical world.

Related papers

• TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models [8.38821563990503]
  Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training.<n>We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models.<n> Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO consistently outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline.
  arXiv  Detail & Related papers  (2026-08-07T15:09:51Z)
• Adapting Generalist Robot Policies with Semantic Reinforcement Learning [59.009122725071386]
  Generalist robot policies learn a diverse repertoire of behaviors from large-scale pretraining.<n>We propose Semantic Action Reinforcement Learning (SARL) to learn to optimize this prompt space through online interaction.<n>We show SARL unlocks fundamentally new capabilities -- adapting VLA behavior to solve complex, long-horizon tasks.
  arXiv  Detail & Related papers  (2026-06-30T17:00:33Z)
• EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models [84.73890225707264]
  We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies.<n>Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data.<n>We release an open-source robustness with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.
  arXiv  Detail & Related papers  (2026-05-25T06:31:03Z)
• Learning while Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies [23.266003019334438]
  Generalist robot policies increasingly benefit from large-scale pretraining, but offline data alone is insufficient for robust real-world deployment.<n>We present Learning While Deploying (LWD), a fleet-scale offline-to-online reinforcement learning framework for continual post-training of generalist Vision-Language-Action (VLA) policies.
  arXiv  Detail & Related papers  (2026-05-01T05:20:26Z)
• DexHiL: A Human-in-the-Loop Framework for Vision-Language-Action Model Post-Training in Dexterous Manipulation [14.050551792714083]
  Human-in-the-Loop (HiL) learning has proven to be a powerful mechanism for refining robot policies.<n>We present DexHiL, the first integrated arm-hand human-in-the-loop framework for dexterous VLA models.
  arXiv  Detail & Related papers  (2026-03-10T02:55:27Z)
• AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models [9.608633915316252]
  Vision-Language-Action (VLA) models demonstrate remarkable potential for generalizable robotic manipulation.<n>Current paradigms rely on coarse, high-level task instructions during supervised fine-tuning.<n>We propose method, the first subtask-aware VLA framework integrated with a scalable offline post-training pipeline.
  arXiv  Detail & Related papers  (2026-03-09T15:52:48Z)
• RLinf-USER: A Unified and Extensible System for Real-World Online Policy Learning in Embodied AI [44.06975079994439]
  We present USER, a Unified and SystEm for Real-world online policy learning.<n>We show that USER enables multi-robot coordination, heterogeneous manipulators, edge-cloud collaboration with large models, and long-running asynchronous training.<n>Results in both simulation and the real world show that USER enables online imitation or reinforcement learning of CNN/MLP, generative policies, and large vision-action (VLA) models within a unified pipeline.
  arXiv  Detail & Related papers  (2026-02-08T06:23:43Z)
• On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning [61.38273866396522]
  Vision-Language-Action models have emerged as a powerful paradigm for general-purpose robot learning.<n>Current methods remain unsuitable for challenging simulated- or physical-world deployments.<n>We introduce a Test-Time Reinforcement Learning for VLAs framework that enables on-the-fly policy adaptation during inference.
  arXiv  Detail & Related papers  (2026-01-11T01:51:30Z)
• WMPO: World Model-based Policy Optimization for Vision-Language-Action Models [22.01666177489494]
  Vision-Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation.<n>We introduce World-Model-based Policy Optimization (WMPO), a principled framework for on-policy VLA without interacting with the real environment.
  arXiv  Detail & Related papers  (2025-11-12T17:54:09Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• PILOT: A Pre-Trained Model-Based Continual Learning Toolbox [65.57123249246358]
  This paper introduces a pre-trained model-based continual learning toolbox known as PILOT.<n>On the one hand, PILOT implements some state-of-the-art class-incremental learning algorithms based on pre-trained models, such as L2P, DualPrompt, and CODA-Prompt.<n>On the other hand, PILOT fits typical class-incremental learning algorithms within the context of pre-trained models to evaluate their effectiveness.
  arXiv  Detail & Related papers  (2023-09-13T17:55:11Z)
• Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
  We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
  arXiv  Detail & Related papers  (2023-06-06T02:24:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.