TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2608.07314v1
• Date: Fri, 07 Aug 2026 15:09:51 GMT
• Title: TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models
• Abstract summary: Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training.<n>We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models.<n> Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO consistently outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline.
• Score: 8.38821563990503
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training. SFT is prone to distribution mismatch, and existing RL approaches typically apply a single, uniform update strategy to all model components, ignoring their distinct functional roles. We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. TEMPO freezes the pretrained vision-language backbone to preserve general semantic representations, and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert. We update them at different rates--the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction. This decoupling RL fine-tuning strategy prevents fast policy updates from destabilizing high-level semantic representations while still allowing the action expert to learn efficiently from online feedback. Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO consistently outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline, while reaching and maintaining higher evaluation rewards on two real-world tasks.

Related papers

• RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models [8.012549967082832]
  $RL2$ is an adaptive inference-time steering framework for Vision-Language-Action (VLA) models.<n>We train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert.<n>This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL.
  arXiv  Detail & Related papers  (2026-07-29T14:49:56Z)
• LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning [90.86828952599147]
  We present LaST-R1, a novel reinforcement learning framework designed to harness "latent reasoning-before-acting" policies.<n>LaST-R1 achieves a near-perfect 99.9% average success rate on the LIBERO benchmark.<n>In real-world deployments, LaST-R1 yields up to a 22.5% average improvement over SOTA supervised fine-tuning approach.
  arXiv  Detail & Related papers  (2026-04-30T17:59:52Z)
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models [51.45122910254346]
  Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models beyond the training domain.<n>Supervised fine-tuning (SFT) frequently leads to general forgetting capabilities.<n>We present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup.
  arXiv  Detail & Related papers  (2026-04-27T21:22:34Z)
• Jump-Start Reinforcement Learning with Vision-Language-Action Regularization [1.2599533416395767]
  Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation.<n>Current limitations hinder their direct use in fast and precise manipulation.<n>We propose Vision-Language-Action Jump-Starting (VLAJS) to improve exploration and learning efficiency.
  arXiv  Detail & Related papers  (2026-04-15T11:17:54Z)
• On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning [61.38273866396522]
  Vision-Language-Action models have emerged as a powerful paradigm for general-purpose robot learning.<n>Current methods remain unsuitable for challenging simulated- or physical-world deployments.<n>We introduce a Test-Time Reinforcement Learning for VLAs framework that enables on-the-fly policy adaptation during inference.
  arXiv  Detail & Related papers  (2026-01-11T01:51:30Z)
• SOP: A Scalable Online Post-Training System for Vision-Language-Action Models [15.86316960521611]
  Vision-language-action (VLA) models achieve strong generalization through large-scale pre-training, but real-world deployment requires expert-level task proficiency in addition to broad generality.<n>Existing post-training approaches for VLA models are typically offline, single-robot, or task-specific.<n>We introduce a Scalable Online Post-training system that enables online, distributed, multi-task post-training of generalist VLA models directly in the physical world.
  arXiv  Detail & Related papers  (2026-01-06T14:25:11Z)
• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach [78.4812458793128]
  We propose textbfTACO, a test-time-scaling framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks.<n>Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits.
  arXiv  Detail & Related papers  (2025-12-02T14:42:54Z)
• Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter [52.111923076688505]
  Training Large Language Models (LLMs) with strong reasoning capabilities marks a significant milestone, unlocking new frontiers in complex problem-solving.<n>We propose TLT, a system that accelerates reasoning RL training losslessly by integrating adaptive speculative decoding.
  arXiv  Detail & Related papers  (2025-11-20T18:59:25Z)
• ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation [57.399685080574756]
  Existing MLLM-based VLN methods rely on imitation learning (IL) and often use DAgger for post-training.<n>We propose ActiveVLN, a VLN framework that explicitly enables active exploration through multi-turn RL.<n>Experiments show that ActiveVLN achieves the largest performance gains over IL baselines compared to both DAgger-based and prior RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-09-16T03:31:46Z)
• ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning [68.76048244253582]
  We introduce ViaRL, the first framework to leverage rule-based reinforcement learning (RL) for optimizing frame selection in video understanding.<n>ViaRL utilizes the answer accuracy of a downstream model as a reward signal to train a frame selector through trial-and-error.<n>ViaRL consistently delivers superior temporal grounding performance and robust generalization across diverse video understanding tasks.
  arXiv  Detail & Related papers  (2025-05-21T12:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.