SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation

• URL: http://arxiv.org/abs/2606.10305v1
• Date: Tue, 09 Jun 2026 01:46:23 GMT
• Title: SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
• Abstract summary: Fine-tuning vision-language-action (VLA) policies for long-horizon manipulation still relies heavily on behavior cloning.<n>We introduce RM, a multi-task stage-aware reward model that combines an action-primitive-based stage estimator with a multi-gate Mixture-of-Experts (MMoE) value head.<n>Building on RM, we further propose SPIRAL, an on-policy reward-guided framework that improves VLA policies from cheap autonomous rollouts.
• Score: 57.131411215593744
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning vision-language-action (VLA) policies for long-horizon manipulation still relies heavily on behavior cloning, which requires costly high-quality demonstrations and keeps policies near the demonstration distribution. Reward models can reduce this dependence by reweighting demonstrations and providing dense supervision for on-robot reinforcement learning (RL), but they must be dense, accurate, and general. Existing methods fall short: task-specific stage-aware models are accurate but require per-task annotations, while general vision-language-model (VLM) reward models are broadly applicable but too coarse for fine-grained long-horizon progress. We introduce RM, a multi-task stage-aware reward model that combines an action-primitive-based stage estimator with a multi-gate Mixture-of-Experts (MMoE) value head to produce dense per-step rewards across manipulation tasks. Building on RM, we further propose SPIRAL (Self-Policy Improvement via Reward-Aligned Learning), an on-policy reward-guided framework that improves VLA policies from cheap autonomous rollouts. On a 10-task benchmark, RM reduces value-estimation MSE by 80% over the strongest baselines; when used in SPIRAL, it improves task success from around 50% to near-perfect performance on Folding Shorts (58% to 100%) and Cleaning Whiteboard (50% to 90%), showing that high-quality dense rewards are key to a stable robot data flywheel. Project website: https://qianzhong-chen.github.io/sarm2.github.io/.

Related papers

• Robust Bimanual Vision-Language-Action Models via Embarrassingly Simple Modality Masking [48.85795530951384]
  We show that structured training-time masking can improve robustness of query-based VLA policies for bimanual manipulation.<n>Results suggest that structured training-time masking is a practical way to improve the robustness of query-based VLA policies for bimanual manipulation.
  arXiv  Detail & Related papers  (2026-08-23T13:48:57Z)
• ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generation [6.976481623762446]
  textttReactVLA is a lightweight and low-latency VLA framework for real-time robotic manipulation.<n>textttReactVLA consistently outperforms similarly sized VLA baselines.<n>It reduces real-world policy latency to below 38.6 ms, enabling fast reactive control on physical robot platforms.
  arXiv  Detail & Related papers  (2026-06-12T08:33:37Z)
• Generalizable Dense Reward for Long-Horizon Robotic Tasks [10.301325532021218]
  Vision-Language Models (VLMs) decompose tasks into verifiable subtasks.<n>Self-certainty provides per-step intrinsic guidance throughout PPO finetuning.<n>On the CHORES benchmark,VLLR achieves up to 56% absolute success rate gains over the pretrained policy.
  arXiv  Detail & Related papers  (2026-03-31T02:05:07Z)
• TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics [46.912038830356714]
  We introduce TOPReward, a novel, probabilistically grounded temporal value function that estimates robotic task progress.<n>In zero-shot evaluations across 130+ distinct real-world tasks, TOPReward achieves 0.947 mean Value-Order Correlation (VOC) on Qwen3-VL.<n>We demonstrate that TOPReward serves as a versatile tool for downstream applications, including success detection and reward-aligned behavior cloning.
  arXiv  Detail & Related papers  (2026-02-22T19:25:48Z)
• RoboReward: General-Purpose Vision-Language Reward Models for Robotics [124.34685604054312]
  Vision-language models (VLMs) have shown promise as automatic reward models, yet their effectiveness on real robot tasks is poorly understood.<n>We aim to close this gap by introducing RoboReward, a robotics reward dataset and benchmark built on large-scale real-robot corpora.
  arXiv  Detail & Related papers  (2026-01-02T12:47:34Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• GRAPE: Generalizing Robot Policy via Preference Alignment [58.419992317452376]
  We present GRAPE: Generalizing Robot Policy via Preference Alignment.<n>We show GRAPE increases success rates on in-domain and unseen manipulation tasks by 51.79% and 58.20%, respectively.<n> GRAPE can be aligned with various objectives, such as safety and efficiency, reducing collision rates by 37.44% and rollout step-length by 11.15%, respectively.
  arXiv  Detail & Related papers  (2024-11-28T18:30:10Z)
• Affordance-Guided Reinforcement Learning via Visual Prompting [51.361977466993345]
  Keypoint-based Affordance Guidance for Improvements (KAGI) is a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL.<n>On diverse real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 30K online fine-tuning steps.
  arXiv  Detail & Related papers  (2024-07-14T21:41:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.