Reinforcement Learning for Real-Time Vision-Language-Action Policies
Abstract Overview
This paper addresses reinforcement learning fine-tuning for large vision-language-action (VLA) policies operating under non-negligible inference latency, which otherwise causes distribution shift from stale observations. The proposed method, Real-Time EXPO-FT, decouples slow asynchronous action-chunk proposals generated by a pretrained VLA from fast reactive action adjustments conditioned on the latest observation. A learned critic evaluates both original and edited candidate action chunks immediately before execution to select the highest-value action sequence. The framework is validated in simulation on the Kinetix benchmark and across four dynamic real-world manipulation tasks requiring high-frequency reactivity.
Novelty
The main novelty is an RL fine-tuning framework for real-time VLA control that explicitly incorporates inference latency rather than assuming instantaneous execution. It combines asynchronous base-action generation via training-time real-time chunking with lightweight, synchronous observation-conditioned action editing and critic-based selection at execution time.
Results
On the Kinetix simulation benchmark, Real-Time EXPO-FT achieves an average success rate of 96.2% under a 4-step inference delay, outperforming all delayed RL baselines across 10 out of 10 environments as well as the zero-delay RLPD baseline (81.4%). In real-world dynamic tasks with online interaction capped at 10 minutes per task, it elevates average task performance from 42% (12.5/30) to 97% (29/30), maintaining robustness across varying latency levels and dynamic object speeds.
Key Points
- Real-Time EXPO-FT prevents stale-observation degradation in large VLA policies by decoupling slow asynchronous chunk generation from fast, observation-conditioned action editing at execution time.
- A chunk-level critic evaluates base and edited candidate actions using temporal-difference learning, maintaining Markovian properties despite inference delays and optionally filtering candidate noise to minimize computation.
- Across 10 simulation tasks and 4 dynamic real-world tasks (dynamic picking, ball balancing, object passing, and soccer kicking), Real-Time EXPO-FT outperforms prior real-time and RL fine-tuning methods within 10 minutes of online robot data.