HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learning

• URL: http://arxiv.org/abs/2606.22860v1
• Date: Mon, 22 Jun 2026 05:07:08 GMT
• Title: HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learning
• Abstract summary: We introduce a novel, plug-and-play fine-tuning pipeline for Vision-Language-Action (VLA) models.<n>We conceptualize VLA-generated actions as a unified interface, upon which we train a residual policy.<n>Within only 1.5 hour of real-world online RL training, the average success rate exceeds 95% on real robots.
• Score: 15.872908522637957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in generative imitation learning have significantly propelled the field of robotic manipulation. However, the majority of existing models rely heavily on Behavior Cloning (BC), a paradigm that suffers from compounding errors and distributional shift. Consequently, the efficacy of these models in practical industrial deployments remains limited. To address these challenges, we introduce a novel, plug-and-play fine-tuning pipeline designed to facilitate the robust deployment of Vision-Language-Action (VLA) models in real-world environments. In contrast to contemporary reinforcement learning (RL) fine-tuning strategies, which are often constrained by specific model architectures, our proposed framework is model-agnostic and adaptable to a diverse range of VLA models. We conceptualize VLA-generated actions as a unified interface, upon which we train a residual policy. This policy is designed to rectify suboptimal actions and address the distributional shift inherent in imitation learning. Additionally, we incorporate human-in-the-loop guidance to ensure safe exploration and maximize training efficiency. We conduct experiments directly in real-world robotic settings. The results demonstrate that within only 1.5 hour of real-world online RL training, the average success rate exceeds 95% on real robots. Our work presents a practical solution for deploying behavior cloning models in industrial scenarios.

Related papers

• EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models [84.73890225707264]
  We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies.<n>Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data.<n>We release an open-source robustness with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.
  arXiv  Detail & Related papers  (2026-05-25T06:31:03Z)
• VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model [87.75549463328836]
  The goal of this paper is to improve the performance and reliability of vision-language-action (VLA) models through iterative online interaction.<n>We propose a simple iterative improvement algorithm that uses real-world roll-out data to improve the fidelity of the world model.<n>We achieve a 39.2% absolute success rate improvement over the base policy and 11.6% improvement from training with the generated synthetic rollouts.
  arXiv  Detail & Related papers  (2026-02-12T15:21:47Z)
• World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policy [55.03832008486675]
  World-VLA-Loop is a closed-loop framework for the joint refinement of world models and Vision-Language-Action (VLA) policies.<n>We propose a state-aware video world model that functions as a high-fidelity interactive simulator by jointly predicting future observations and reward signals.
  arXiv  Detail & Related papers  (2026-02-06T08:57:55Z)
• Reshaping Action Error Distributions for Reliable Vision-Language-Action Models [69.38615670891038]
  In robotic manipulation, vision-language-action (VLA) models have emerged as a promising paradigm for learning generalizable and scalable robot policies.<n>We focus on continuous-action VLA models and move beyond conventional MSE-based regression by reshaping action error distributions during training.<n>We evaluate our approaches across standard, few-shot, and noisy settings on multiple representative VLA architectures.
  arXiv  Detail & Related papers  (2026-02-04T05:37:09Z)
• Towards Accessible Physical AI: LoRA-Based Fine-Tuning of VLA Models for Real-World Robot Control [0.0]
  This paper presents an efficient fine-tuning methodology and real-world deployment analysis for adapting VLA models to low-cost robotic manipulation systems.<n>We propose a resource-efficient fine-tuning strategy using Low-Rank Adaptation (LoRA) and quantization techniques.<n>Our methodology addresses the critical challenge of adapting pre-trained VLA models to new robot embodiments with limited demonstration data.
  arXiv  Detail & Related papers  (2025-12-11T16:25:30Z)
• Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance [63.33213516925946]
  We introduce textbfAlign-Then-stEer (textttATE), a novel, data-efficient, and plug-and-play adaptation framework.<n>Our work presents a general and lightweight solution that greatly enhances the practicality of deploying VLA models to new robotic platforms and tasks.
  arXiv  Detail & Related papers  (2025-09-02T07:51:59Z)
• Human-assisted Robotic Policy Refinement via Action Preference Optimization [26.144183856600687]
  Action Preference Optimization (APO) is a method designed to refine Vision-Language-Action (VLA) models by human-assisted preference alignment.<n>To solve this, APO proposes an adaptive reweighting algorithm with binary desirability signals derived from interaction.<n>Experiments conducted in simulation and real-world scenarios prove superior generalization and robustness.
  arXiv  Detail & Related papers  (2025-06-08T13:14:18Z)
• VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning [14.099306230721245]
  We present VLA-RL, an exploration-based framework that improves on online collected data at test time.<n>We fine-tune a pretrained vision-language model as a robotic process reward model, which is trained on pseudo reward labels annotated on automatically extracted task segments.<n>VLA-RL enables OpenVLA-7B to surpass the strongest finetuned baseline by 4.5% on 40 challenging robotic manipulation tasks in LIBERO.
  arXiv  Detail & Related papers  (2025-05-24T14:42:51Z)
• Offline Robotic World Model: Learning Robotic Policies without a Physics Simulator [50.191655141020505]
  Reinforcement Learning (RL) has demonstrated impressive capabilities in robotic control but remains challenging due to high sample complexity, safety concerns, and the sim-to-real gap.<n>We introduce Offline Robotic World Model (RWM-O), a model-based approach that explicitly estimates uncertainty to improve policy learning without reliance on a physics simulator.
  arXiv  Detail & Related papers  (2025-04-23T12:58:15Z)
• Reinforcement Learning for Machine Learning Model Deployment: Evaluating Multi-Armed Bandits in ML Ops Environments [0.0]
  We investigate whether reinforcement learning (RL)-based model management can manage deployment decisions more effectively.<n>Our approach enables more adaptive production environments by continuously evaluating deployed models and rolling back underperforming ones in real-time.<n>Our findings suggest that RL-based model management can improve automation, reduce reliance on manual interventions, and mitigate risks associated with post-deployment model failures.
  arXiv  Detail & Related papers  (2025-03-28T16:42:21Z)
• ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy [10.596344084789434]
  Vision-Language-Action (VLA) models have shown substantial potential in real-world robotic manipulation.<n>Fine-tuning these models through supervised learning struggles to achieve robust performance due to limited, inconsistent demonstrations.<n>We propose a reinforced fine-tuning approach for VLA models, named ConRFT, which consists of offline and online fine-tuning.
  arXiv  Detail & Related papers  (2025-02-08T05:01:17Z)
• Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics [50.191655141020505]
  This work advances model-based reinforcement learning by addressing the challenges of long-horizon prediction, error accumulation, and sim-to-real transfer.<n>By providing a scalable and robust framework, the introduced methods pave the way for adaptive and efficient robotic systems in real-world applications.
  arXiv  Detail & Related papers  (2025-01-17T10:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.