RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards

• URL: http://arxiv.org/abs/2606.26175v1
• Date: Wed, 24 Jun 2026 10:32:48 GMT
• Title: RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards
• Abstract summary: Reinforcement learning for robotic manipulation often requires manually designing a dense reward function.<n>We argue that a single global prompt is too coarse for long-horizon manipulation tasks with randomized initial conditions.<n>We propose Reinforced Micro-Task Learning (RMTL), an approach that decomposes a manipulation task into a small set of language-described micro-tasks.
• Score: 0.688204255655161
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) for robotic manipulation often requires manually designing a dense reward function, which is difficult to tune and often fragile, or learning a reward from human demonstrations or preferences, which can be expensive. A recent line of work uses pretrained vision-language models (VLMs) as zero-shot reward models, replacing these costs with a single text prompt. However, we argue that a single global prompt is too coarse for long-horizon manipulation tasks with randomized initial conditions. The single-prompt VLM reward is near-flat for much of the trajectory, making early progress hard for the agent to detect. We propose Reinforced Micro-Task Learning (RMTL), an approach that decomposes a manipulation task into a small set of language-described micro-tasks and trains the agent to switch between them. At each step, the agent receives a multi-view VLM reward computed using the prompt of the currently active micro-task and averaged across multiple camera views to reduce the effect of view-specific occlusions. A reverse curriculum gradually exposes the agent to harder initial conditions, while a PPO worker is first trained with a fixed distance-based rule that selects the active micro-task. We then replace this rule with a learned hierarchical manager, turning rule-based phase selection into a fully learned hierarchical policy. We instantiate RMTL on the Fetch manipulation environment using three short stage-specific prompts and without additional prompt tuning. Experiments show that RMTL provides more informative reward signals than single-prompt VLM rewards, enabling faster learning. These results suggest that decomposing VLM rewards into micro-task-specific language prompts can substantially improve the scalability of language-guided reinforcement learning for robotic manipulation.

Related papers

• SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation [57.131411215593744]
  Fine-tuning vision-language-action (VLA) policies for long-horizon manipulation still relies heavily on behavior cloning.<n>We introduce RM, a multi-task stage-aware reward model that combines an action-primitive-based stage estimator with a multi-gate Mixture-of-Experts (MMoE) value head.<n>Building on RM, we further propose SPIRAL, an on-policy reward-guided framework that improves VLA policies from cheap autonomous rollouts.
  arXiv  Detail & Related papers  (2026-06-09T01:46:23Z)
• Generalizable Dense Reward for Long-Horizon Robotic Tasks [10.301325532021218]
  Vision-Language Models (VLMs) decompose tasks into verifiable subtasks.<n>Self-certainty provides per-step intrinsic guidance throughout PPO finetuning.<n>On the CHORES benchmark,VLLR achieves up to 56% absolute success rate gains over the pretrained policy.
  arXiv  Detail & Related papers  (2026-03-31T02:05:07Z)
• GoalLadder: Incremental Goal Discovery with Vision-Language Models [38.35578010611503]
  We propose a novel method to train RL agents from a single language instruction in visual environments.<n>GoalLadder works by incrementally discovering states that bring the agent closer to completing a task specified in natural language.<n>Unlike prior work, GoalLadder does not trust VLM's feedback completely; instead, it uses it to rank potential goal states using an ELO-based rating system.
  arXiv  Detail & Related papers  (2025-06-19T15:28:27Z)
• Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning [49.20643505834273]
  Embodied agents operating in household environments must interpret ambiguous and under-specified human instructions.<n>We propose a novel approach that fine-tunes multi-modal large language models (MLLMs) as vision-language-action (VLA) policies using online reinforcement learning (RL) with LLM-generated rewards.<n>Our results show that our RL-finetuned MLLM outperforms all baselines by a significant margin (10.4-16.5%), generalizing well to novel scenes and tasks.
  arXiv  Detail & Related papers  (2025-04-01T15:41:50Z)
• Reward Machine Inference for Robotic Manipulation [1.6135226672466307]
  Reward Machines (RMs) enhance RL's capability to train policies over extended time horizons.<n>We introduce a novel LfD approach for learning RMs directly from visual demonstrations of robotic manipulation tasks.<n>We validate our method on vision-based manipulation tasks, showing that the inferred RM accurately captures task structure and enables an RL agent to effectively learn an optimal policy.
  arXiv  Detail & Related papers  (2024-12-13T12:32:53Z)
• Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards [49.7719149179179]
  This paper investigates the feasibility of using PPO for reinforcement learning (RL) from explicitly programmed reward signals. We focus on tasks expressed through formal languages, such as programming, where explicit reward functions can be programmed to automatically assess quality of generated outputs. Our results show that pure RL-based training for the two formal language tasks is challenging, with success being limited even for the simple arithmetic task.
  arXiv  Detail & Related papers  (2024-10-22T15:59:58Z)
• CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answering [23.360714576158905]
  Large vision-language models (VLMs) have shown significant performance boost in various application domains. Finetuning a VLM on a task leads to reducing its generalization power and the capacity of learning new tasks. We propose a novel prompt-based CL method forVLMs, namely $textbfClu$ster-based $textbfMo$dality Fusion Prompt (textbfCluMo)
  arXiv  Detail & Related papers  (2024-08-21T16:07:49Z)
• Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning [12.628697648945298]
  Reinforcement learning (RL) requires either manually specifying a reward function, or learning a reward model from a large amount of human feedback. We study a more sample-efficient alternative: using pretrained vision-language models (VLMs) as zero-shot reward models (RMs) to specify tasks via natural language.
  arXiv  Detail & Related papers  (2023-10-19T17:17:06Z)
• Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning [49.92517970237088]
  We tackle the problem of training a robot to understand multimodal prompts. This type of task poses a major challenge to robots' capability to understand the interconnection and complementarity between vision and language signals. We introduce an effective framework that learns a policy to perform robot manipulation with multimodal prompts.
  arXiv  Detail & Related papers  (2023-10-14T22:24:58Z)
• Language Reward Modulation for Pretraining Reinforcement Learning [61.76572261146311]
  We propose leveraging the capabilities of LRFs as a pretraining signal for reinforcement learning. Our VLM pretraining approach, which is a departure from previous attempts to use LRFs, can warmstart sample-efficient learning on robot manipulation tasks.
  arXiv  Detail & Related papers  (2023-08-23T17:37:51Z)
• Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting [83.21164539349273]
  Pre-trained language models (PLMs) have played an increasing role in multimedia research. In this paper, we focus on exploring PLMs as a stand-alone model for vision-language reasoning tasks. We propose a novel transfer learning approach for PLMs, termed Dynamic Visual Prompting (DVP)
  arXiv  Detail & Related papers  (2023-06-01T07:19:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.