FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

• URL: http://arxiv.org/abs/2606.26006v1
• Date: Wed, 24 Jun 2026 16:23:18 GMT
• Title: FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation
• Abstract summary: FORCE is a 3-stage framework that stabilizes fine-tuning by tackling both issues.<n>FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%.
• Score: 40.53898394579726
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.

Related papers

• PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models [50.232333672172395]
  Vision-Language-Action (VLA) models provide a unified paradigm for robotic manipulation, yet their real-world deployment is often bottlenecked by execution efficiency.<n>We propose textbfPolicyTrim, a reinforcement learning-based post-training framework that extends the reliable action chunk length and reduces redundant physical steps.<n>Our framework delivers up to a 5.83$times$ end-to-end deployment speedup without compromising task success rates.
  arXiv  Detail & Related papers  (2026-06-21T14:54:07Z)
• Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes [40.20394793623452]
  When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome.<n>We propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for viability and efficiency.<n>HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.
  arXiv  Detail & Related papers  (2026-06-15T17:57:14Z)
• Q-VGM: Q-Guided Value-Gradient Matching for Flow-Matching VLA Policies [14.519898493996891]
  We propose Q-Guided Value-Gradient Matching (Q-VGM) as an off-policy reinforcement learning (RL) method.<n>Q-VGM sidesteps issues by leveraging VGG-Flow, a value-gradient view of flow alignment in generative modeling.<n>On LIBERO, Q-VGM raises the average success rate from 75.0% to 92.5%; on RoboTwin 2.0, from 76.4% to 87.2%; and on two real-robot tabletop tasks, from 40.0% to 67.5%.
  arXiv  Detail & Related papers  (2026-06-06T07:10:25Z)
• CF-VLA: Efficient Coarse-to-Fine Action Generation for Vision-Language-Action Policies [26.36935074374452]
  Flow-based vision-language-action (VLA) policies offer strong expressivity for action generation, but suffer from a fundamental inefficiency.<n>We propose CF-VLA, a coarse-to-fine two-stage formulation that restructures action generation into a coarse-to-fine step.<n>We show that our method establishes a strong efficiency-performance frontier under low-NFE (Number of Function Evaluations) regimes.
  arXiv  Detail & Related papers  (2026-04-27T15:51:40Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• SG-OIF: A Stability-Guided Online Influence Framework for Reliable Vision Data [6.4391040754741296]
  In this paper, we introduce a Stability-Guided Online Influence Framework (SG-OIF) for Approximating training-point influence on test predictions.<n>We show that SG-OIF achieves 91.1% accuracy in the top 1% prediction samples on the CIFAR-10, and 99.8% AUPR score on MNIST.
  arXiv  Detail & Related papers  (2025-11-21T19:58:54Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization [53.82400605816587]
  Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.<n>A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios.<n>We introduce Continual AQA (CAQA), which equips with Continual Learning capabilities to handle evolving distributions.
  arXiv  Detail & Related papers  (2025-10-08T10:09:47Z)
• A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning [54.48409201256968]
  offline-to-online Reinforcement Learning (O2O RL) aims to improve the performance of offline pretrained policy using only a few online samples. Most O2O methods focus on the balance between RL objective and pessimism, or the utilization of offline and online samples.
  arXiv  Detail & Related papers  (2023-12-12T19:24:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.