Policy Learning from Large Vision-Language Model Feedback without Reward Modeling

• URL: http://arxiv.org/abs/2507.23391v1
• Date: Thu, 31 Jul 2025 10:07:49 GMT
• Title: Policy Learning from Large Vision-Language Model Feedback without Reward Modeling
• Authors: Tung M. Luu, Donghoon Lee, Younghwan Lee, Chang D. Yoo,
• Abstract summary: We introduce PLARE, a novel approach that leverages large vision-language models (VLMs) to provide guidance signals for agent training.<n>Instead of relying on manually designed reward functions, PLARE queries a VLM for preference labels on pairs of visual trajectory segments.<n>The policy is then trained directly from these preference labels using a supervised contrastive preference learning objective.
• Score: 19.48826538310603
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline reinforcement learning (RL) provides a powerful framework for training robotic agents using pre-collected, suboptimal datasets, eliminating the need for costly, time-consuming, and potentially hazardous online interactions. This is particularly useful in safety-critical real-world applications, where online data collection is expensive and impractical. However, existing offline RL algorithms typically require reward labeled data, which introduces an additional bottleneck: reward function design is itself costly, labor-intensive, and requires significant domain expertise. In this paper, we introduce PLARE, a novel approach that leverages large vision-language models (VLMs) to provide guidance signals for agent training. Instead of relying on manually designed reward functions, PLARE queries a VLM for preference labels on pairs of visual trajectory segments based on a language task description. The policy is then trained directly from these preference labels using a supervised contrastive preference learning objective, bypassing the need to learn explicit reward models. Through extensive experiments on robotic manipulation tasks from the MetaWorld, PLARE achieves performance on par with or surpassing existing state-of-the-art VLM-based reward generation methods. Furthermore, we demonstrate the effectiveness of PLARE in real-world manipulation tasks with a physical robot, further validating its practical applicability.

Related papers

• LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback [121.78866929908871]
  Large Action Models (LAMs) for AI Agents offer incredible potential but face challenges due to the need for high-quality training data.<n>We present LAM SIMULATOR, a comprehensive framework designed for online exploration of agentic tasks with high-quality feedback.<n>Our framework features a dynamic task query generator, an extensive collection of tools, and an interactive environment where Large Language Model (LLM) Agents can call tools and receive real-time feedback.
  arXiv  Detail & Related papers  (2025-06-02T22:36:02Z)
• Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning [19.48826538310603]
  In offline reinforcement learning (RL), learning from fixed datasets presents a promising solution for domains where real-time interaction with the environment is expensive or risky.<n>We propose Reward Generation via Large Vision-Language Models (RG-VLM) to generate rewards from offline data without human involvement.
  arXiv  Detail & Related papers  (2025-04-03T07:11:18Z)
• ViVa: Video-Trained Value Functions for Guiding Online RL from Diverse Data [56.217490064597506]
  We propose and analyze a data-driven methodology that automatically guides RL by learning from widely available video data.<n>We use intent-conditioned value functions to learn from diverse videos and incorporate these goal-conditioned values into the reward.<n>Our experiments show that video-trained value functions work well with a variety of data sources, exhibit positive transfer from human video pre-training, can generalize to unseen goals, and scale with dataset size.
  arXiv  Detail & Related papers  (2025-03-23T21:24:33Z)
• Real-World Offline Reinforcement Learning from Vision Language Model Feedback [19.494335952082466]
  offline reinforcement learning can enable policy learning from pre-collected, sub-optimal datasets without online interactions. Most existing offline RL works assume the dataset is already labeled with the task rewards. We propose a novel system that automatically generates reward labels for offline datasets.
  arXiv  Detail & Related papers  (2024-11-08T02:12:34Z)
• Affordance-Guided Reinforcement Learning via Visual Prompting [51.361977466993345]
  Keypoint-based Affordance Guidance for Improvements (KAGI) is a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL.<n>On diverse real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 30K online fine-tuning steps.
  arXiv  Detail & Related papers  (2024-07-14T21:41:29Z)
• RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback [24.759613248409167]
  Reward engineering has long been a challenge in Reinforcement Learning research. We propose RL-VLM-F, a method that automatically generates reward functions for agents to learn new tasks. We demonstrate that RL-VLM-F successfully produces effective rewards and policies across various domains.
  arXiv  Detail & Related papers  (2024-02-06T04:06:06Z)
• Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning [58.3994826169858]
  We introduce RoboFuME, a reset-free fine-tuning system for robotic reinforcement learning. Our insights are to utilize offline reinforcement learning techniques to ensure efficient online fine-tuning of a pre-trained policy. Our method can incorporate data from an existing robot dataset and improve on a target task within as little as 3 hours of autonomous real-world experience.
  arXiv  Detail & Related papers  (2023-10-23T17:50:08Z)
• Leveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environments [4.2569494803130565]
  We introduce an innovative algorithm designed to assign rewards to offline trajectories, using a small number of high-quality expert demonstrations. This approach circumvents the need for handcrafted rewards, unlocking the potential to harness vast datasets for policy learning.
  arXiv  Detail & Related papers  (2023-10-13T03:39:15Z)
• Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
  We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
  arXiv  Detail & Related papers  (2023-06-06T02:24:41Z)
• Offline Reinforcement Learning from Images with Latent Space Models [60.69745540036375]
  offline reinforcement learning (RL) refers to the problem of learning policies from a static dataset of environment interactions. We build on recent advances in model-based algorithms for offline RL, and extend them to high-dimensional visual observation spaces. Our approach is both tractable in practice and corresponds to maximizing a lower bound of the ELBO in the unknown POMDP.
  arXiv  Detail & Related papers  (2020-12-21T18:28:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.