Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning

• URL: http://arxiv.org/abs/2608.25350v1
• Date: Wed, 26 Aug 2026 04:09:21 GMT
• Title: Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning
• Abstract summary: Vision-language models (VLMs) have emerged as a powerful source of supervision for reinforcement learning.<n>In this work, we introduce the first framework that combines VLM-generated preferences with the Plackett-Luce model for reward learning.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) have emerged as a powerful source of supervision for reinforcement learning, enabling agents to leverage rich semantic knowledge during training. Inspired by the success of preference-based reward learning (PbRL) in reinforcement learning from human feedback (RLHF), vision-language model generated image-based preferences provide an effective source for learning reward functions. This can be done by visually comparing two outcomes through the Bradley-Terry (BT) model. However, this pairwise formulation utilizes only two observations at a time, despite VLMs being capable of ranking multiple candidates. The Plackett-Luce (PL) formulation can shape a reward model with listwise rankings as opposed to pairwise preferences, allowing for a more suited use of a VLM based ranking. In this work, to our knowledge, we introduce the first framework that combines VLM-generated preferences with the Plackett-Luce model for reward learning. We evaluate our approach on Meta-World manipulation tasks and show that Plackett-Luce (PL) reward models can train robotic policies from VLM-generated rankings as effectively as pairwise Bradley-Terry, $K$-wise Bradley-Terry, and RL-VLM-F baselines. Across all environments, at least one PL ranking size ($K \in \{3,4,5\}$) consistently performs with or outperforms other methods in mean success rate. Unlike pairwise methods, which are restricted to $K=2$, PL supports different ranking sizes and can therefore be adapted to the environment and desired feedback format. Our best PL configuration achieves an 86% mean final success rate and matches the Oracle baseline on Drawer Open. Overall, these results demonstrate that listwise VLM preference supervision is a competitive and flexible approach to reward learning for reinforcement learning.

Related papers

• Automating Potential-based Reward Shaping with Vision Language Model Guidance [2.8360662552057323]
  We introduce the VLM-guided PBRS framework that learns the potential function directly from vision language model feedback.<n>As this approach is based on potential-based reward shaping, it preserves the original optimal policies.<n>Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning.
  arXiv  Detail & Related papers  (2026-06-25T15:45:57Z)
• Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling [22.480423145082856]
  We propose Ranked Choice Preference Optimization (RCPO), a unified framework that bridges preference optimization with (ranked) choice modeling.<n>We show how directly leveraging ranked preference data, combined with the right choice models, yields more effective alignment.
  arXiv  Detail & Related papers  (2025-10-24T03:48:47Z)
• Generalist Reward Models: Found Inside Large Language Models [50.7432354447554]
  We show that a powerful reward model is already latently present within any Large Language Models (LLMs) trained via standard next-token prediction.<n>We prove that this endogenous reward is not a reward function learned through offline inverse reinforcement learning.<n>We also prove that subsequent reinforcement learning using this endogenous reward leads to a policy with a provably superior error bound compared to the base model.
  arXiv  Detail & Related papers  (2025-06-29T13:45:54Z)
• Improving LLM General Preference Alignment via Optimistic Online Mirror Descent [57.622821649679786]
  Reinforcement learning from human feedback (RLHF) has demonstrated remarkable effectiveness in aligning large language models (LLMs) with human preferences.<n>In this paper, we drop the Bradley-Terry (BT) model assumption and study LLM alignment under general preferences, formulated as a two-player game.<n>We show that our approach achieves an $O(T-1)$ bound on the duality gap, improving upon the previous $O(T-1/2)$ result.
  arXiv  Detail & Related papers  (2025-02-24T05:24:52Z)
• InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model [80.93387166769679]
  We present IXC-2.5-Reward, a simple yet effective multi-modal reward model that aligns Large Vision Language Models with human preferences.<n>IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks.
  arXiv  Detail & Related papers  (2025-01-21T18:47:32Z)
• Bootstrapping Language Models with DPO Implicit Rewards [45.68366127605774]
  Direct preference optimization (DPO) has greatly simplified the process from past work in reinforcement learning from human feedback.<n>In this work, we make a novel observation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM.<n>Our approach, named self-alignment with DPO ImpliCit rEwards (DICE), shows great improvements in alignment.
  arXiv  Detail & Related papers  (2024-06-14T06:57:18Z)
• Generalizing Reward Modeling for Out-of-Distribution Preference Learning [3.9160947065896803]
  Preference learning with large language models (LLMs) aims to align the LLMs' generations with human preferences. Due to the difficulty of obtaining human feedback, discretely training reward models for every encountered distribution is challenging. This work addresses OOD PL by optimizing a general reward model through a meta-learning approach.
  arXiv  Detail & Related papers  (2024-02-22T18:20:33Z)
• LiPO: Listwise Preference Optimization through Learning-to-Rank [62.02782819559389]
  Policy can learn more effectively from a ranked list of plausible responses given the prompt.<n>We show that LiPO-$lambda$ can outperform DPO variants and SLiC by a clear margin on several preference alignment tasks.
  arXiv  Detail & Related papers  (2024-02-02T20:08:10Z)
• Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models [69.51130760097818]
  We propose Zooter, a reward-guided routing method distilling rewards on training queries to train a routing function. We evaluate Zooter on a comprehensive benchmark collection with 26 subsets on different domains and tasks.
  arXiv  Detail & Related papers  (2023-11-15T04:40:43Z)
• Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons [79.98542868281473]
  We provide a theoretical framework for Reinforcement Learning with Human Feedback (RLHF) We show that when training a policy based on the learned reward model, MLE fails while a pessimistic MLE provides policies with improved performance under certain coverage assumptions.
  arXiv  Detail & Related papers  (2023-01-26T18:07:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.