Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes

• URL: http://arxiv.org/abs/2606.17043v1
• Date: Mon, 15 Jun 2026 17:57:14 GMT
• Title: Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes
• Abstract summary: When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome.<n>We propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for viability and efficiency.<n>HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.
• Score: 40.20394793623452
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome (success or failure), yet the actor update requires per-transition supervision. Existing approaches commonly reduce this sparse outcome to a single scalar reward or advantage signal, which conflates distinct forms of transition-level feedback and provides limited guidance once basic task success becomes achievable. First, a single scalar signal conflates the two objectives of viability and efficiency; once basic success is achieved, the binary label provides no gradient to distinguish efficient completions from slow ones. Second, real-world rollouts mix autonomous and intervention segments; naively assigning episode outcomes across these boundaries introduces incorrect credit assignment. To address these issues, we propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for these two objectives on different data subsets and combines their outputs with a state-adaptive balance. A state-adaptive gate $g_t$ merges their one-step advantages, prioritizing viability when success is uncertain and shifting to efficiency only when viability is high, and converts the result into per-transition weights on the actor loss. Intervention-aware credit assignment further restricts outcome labels to segments executed by the current policy, preventing supervision from leaking across intervention boundaries. In real-robot experiments on three contact-rich bimanual tasks, HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.

Related papers

• TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents [22.01777615106231]
  Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes.<n>We propose Transition-wise Credit Assignment, which derives step-level supervision directly from action-induced transitions.<n>Experiments on ALFWorld, WebShop, and seven search-augmented question-answering benchmarks show consistent improvements over the evaluated baselines.
  arXiv  Detail & Related papers  (2026-08-17T06:19:12Z)
• PACE: Phase-Progress-Aware Credit for Long-Horizon Embodied Manipulation [17.66354304197193]
  Policy improvement requires step-level credit signals to distinguish behaviors that advance the task from those that stall or regress.<n>We present PACE, a credit-assignment framework for post-training on long-horizon manipulation.
  arXiv  Detail & Related papers  (2026-08-15T04:25:16Z)
• Verifier-Induced Support Reshaping in On-Policy Optimization [27.782412748608255]
  We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce.<n>We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget.
  arXiv  Detail & Related papers  (2026-07-31T19:05:43Z)
• FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation [40.53898394579726]
  FORCE is a 3-stage framework that stabilizes fine-tuning by tackling both issues.<n>FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%.
  arXiv  Detail & Related papers  (2026-06-24T16:23:18Z)
• PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models [50.232333672172395]
  Vision-Language-Action (VLA) models provide a unified paradigm for robotic manipulation, yet their real-world deployment is often bottlenecked by execution efficiency.<n>We propose textbfPolicyTrim, a reinforcement learning-based post-training framework that extends the reliable action chunk length and reduces redundant physical steps.<n>Our framework delivers up to a 5.83$times$ end-to-end deployment speedup without compromising task success rates.
  arXiv  Detail & Related papers  (2026-06-21T14:54:07Z)
• PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment [35.978805768172656]
  Long-horizon agentic tasks pose a fundamental credit assignment challenge for outcome-base reinforcement learning.<n>We propose PBSD (Privileged Bayesian Self-Distillation), a Bayes-calibrated self-distillation method for fine-grained credit assignment under sparse final rewards.
  arXiv  Detail & Related papers  (2026-06-08T11:20:58Z)
• Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation [40.17737666526493]
  We propose PACT, a Preference-calibrated Actor-Critic Training framework.<n>We first design a progress model that learns from human demonstration and identifies suboptimal segments for credit correction.<n>We then build preference pairs to define a counterfactual advantage that penalizes Bellman targets of the identified suboptimal segment.
  arXiv  Detail & Related papers  (2026-06-02T17:38:25Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning [6.365332042924078]
  We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution.<n>StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts.<n>Results reveal a consistent two-knob law: smaller _clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength _mix remains task-dependent.
  arXiv  Detail & Related papers  (2026-05-26T15:07:03Z)
• Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models [32.87104367896446]
  We propose an end-to-end framework that leverages failure trajectories as adaptive negative guidance for diffusion- and flow-based VLA policies.<n>AFIL uses a pretrained VLA to generate failure rollouts online, avoiding the need for handcrafted failure-mode design.<n>It then jointly trains Dual Action Generators (DAGs) for successful and failed behaviors while sharing a common vision-language backbone.
  arXiv  Detail & Related papers  (2026-05-08T19:57:11Z)
• TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training [53.93696896939915]
  Training tool-use agents typically rely on Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks.<n>We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology.<n>TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T10:38:54Z)
• Hierarchical Vision Language Action Model Using Success and Failure Demonstrations [60.82332413442677]
  We introduce VINE, a hierarchical vision-language-action model that separates high-level reasoning from low-level control.<n>System 2 performs feasibility-guided tree search over a 2D scene-graph abstraction.<n>System 1 executes low-level actions without modifying the agent's core skills.
  arXiv  Detail & Related papers  (2025-12-03T15:58:38Z)
• Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization [53.82400605816587]
  Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.<n>A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios.<n>We introduce Continual AQA (CAQA), which equips with Continual Learning capabilities to handle evolving distributions.
  arXiv  Detail & Related papers  (2025-10-08T10:09:47Z)
• Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning [71.30276778807068]
  We propose a unified framework that strategically coordinates sample pruning and token pruning.<n>Q-Tuning achieves a +38% average improvement over the full-data SFT baseline using only 12.5% of the original training data.
  arXiv  Detail & Related papers  (2025-09-28T13:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.