Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards

• URL: http://arxiv.org/abs/2503.19948v1
• Date: Tue, 25 Mar 2025 15:30:21 GMT
• Title: Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards
• Authors: Alexander Gambashidze, Konstantin Sobolev, Andrey Kuznetsov, Ivan Oseledets,
• Abstract summary: Using datasets such as ImageReward and Human Preference Score v2 (HPSv2), our models achieve accuracies of 64.9% on the ImageReward test set and 65.4% on HPSv2.<n>Our findings can be a strong mile-stone that will enhance text-to-vision models even further.
• Score: 45.84931291646799
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can Visual Language Models (VLMs) effectively capture human visual preferences? This work addresses this question by training VLMs to think about preferences at test time, employing reinforcement learning methods inspired by DeepSeek R1 and OpenAI O1. Using datasets such as ImageReward and Human Preference Score v2 (HPSv2), our models achieve accuracies of 64.9% on the ImageReward test set (trained on ImageReward official split) and 65.4% on HPSv2 (trained on approximately 25% of its data). These results match traditional encoder-based models while providing transparent reasoning and enhanced generalization. This approach allows to use not only rich VLM world knowledge, but also its potential to think, yielding interpretable outcomes that help decision-making processes. By demonstrating that human visual preferences reasonable by current VLMs, we introduce efficient soft-reward strategies for image ranking, outperforming simplistic selection or scoring methods. This reasoning capability enables VLMs to rank arbitrary images-regardless of aspect ratio or complexity-thereby potentially amplifying the effectiveness of visual Preference Optimization. By reducing the need for extensive markup while improving reward generalization and explainability, our findings can be a strong mile-stone that will enhance text-to-vision models even further.

Related papers

• Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection [53.558449071113245]
  Vision-Language Models (VLMs) leverage aligned visual encoders to transform images into visual tokens, allowing them to be processed similarly to text by the backbone large language model (LLM) Recent advancements in vision-language modeling introduce image cropping techniques that feed all encoded sub-images into the model. We propose a lightweight, universal framework that seamlessly integrates with existing VLMs to enhance their ability to process finegrained details.
  arXiv  Detail & Related papers  (2025-03-14T18:33:31Z)
• From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models [58.16075709485292]
  CAREVL is a novel method for preference reward modeling by reliably using both high- and low-confidence data. CAREVL achieves performance improvements over traditional distillation-based methods on VL-RewardBench and MLLM-as-a-Judge benchmark.
  arXiv  Detail & Related papers  (2025-03-08T16:13:18Z)
• Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images [7.823336661261962]
  Large Vision-Language Models (VLMs) tend to neglect image content and over-rely on language-model priors.<n>We propose S-VCO (Symmetrical Visual Contrastive Optimization), a novel finetuning objective that steers the model toward capturing important visual details.
  arXiv  Detail & Related papers  (2025-02-19T18:05:42Z)
• Probing Visual Language Priors in VLMs [51.016683265437536]
  We introduce ViLP, a benchmark featuring deliberately out-of-distribution images.<n>Each question in ViLP is coupled with three potential answers and three corresponding images.<n>We propose a self-improving framework in which models generate new VQA data, then apply pixel-level and semantic corruptions to form "good-bad" image pairs for self-training.
  arXiv  Detail & Related papers  (2024-12-31T17:54:29Z)
• VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation [70.68566282567207]
  We present VisionReward, a framework for learning human visual preferences in both image and video generation.<n>VisionReward can significantly outperform existing image and video reward models on both machine metrics and human evaluation.
  arXiv  Detail & Related papers  (2024-12-30T16:24:09Z)
• RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data [47.55541945729117]
  Large vision-language models (LVLMs) often fail to align with human preferences. We present a Robust Visual Reward Model (RoVRM) which improves human-preference alignment for LVLMs.
  arXiv  Detail & Related papers  (2024-08-22T03:49:18Z)
• Aligning Modalities in Vision Large Language Models via Preference Fine-tuning [67.62925151837675]
  In this work, we frame the hallucination problem as an alignment issue, tackle it with preference tuning. Specifically, we propose POVID to generate feedback data with AI models. We use ground-truth instructions as the preferred response and a two-stage approach to generate dispreferred data. In experiments across broad benchmarks, we show that we can not only reduce hallucinations, but improve model performance across standard benchmarks, outperforming prior approaches.
  arXiv  Detail & Related papers  (2024-02-18T00:56:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.