ViPO: Visual Preference Optimization at Scale

• URL: http://arxiv.org/abs/2604.24953v2
• Date: Wed, 29 Apr 2026 03:07:25 GMT
• Title: ViPO: Visual Preference Optimization at Scale
• Abstract summary: Current open-source preference datasets contain conflicting preference patterns.<n>We propose Poly-DPO, which dynamically adjusts model confidence based on dataset characteristics.<n>For ViPO, models achieve performance far exceeding those trained on existing open-source preference datasets.
• Score: 16.662106439715238
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While preference optimization is crucial for improving visual generative models, how to effectively scale this paradigm remains largely unexplored. Current open-source preference datasets contain conflicting preference patterns, where winners excel in some dimensions but underperform in others. Naively optimizing on such noisy datasets fails to learn preferences, hindering effective scaling. To enhance robustness against noise, we propose Poly-DPO, which extends the DPO objective with an additional polynomial term that dynamically adjusts model confidence based on dataset characteristics, enabling effective learning across diverse data distributions. Beyond biased patterns, existing datasets suffer from low resolution, limited prompt diversity, and imbalanced distributions. To facilitate large-scale visual preference optimization by tackling data bottlenecks, we construct ViPO, a massive-scale preference dataset with 1M image pairs at 1024px across five categories and 300K video pairs at 720p+ across three categories. State-of-the-art generative models and diverse prompts ensure reliable preference signals with balanced distributions. Remarkably, when applying Poly-DPO to our high-quality dataset, the optimal configuration converges to standard DPO. This convergence validates dataset quality and Poly-DPO's adaptive nature: sophisticated optimization becomes unnecessary with sufficient data quality, yet remains valuable for imperfect datasets. We validate our approach across visual generation models. On noisy datasets like Pick-a-Pic V2, Poly-DPO achieves 6.87 and 2.32 gains over Diffusion-DPO on GenEval for SD1.5 and SDXL, respectively. For ViPO, models achieve performance far exceeding those trained on existing open-source preference datasets. These results confirm that addressing both algorithmic adaptability and data quality is essential for scaling visual preference optimization.

Related papers

• Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training [36.73565681027731]
  We propose Data-DPO, a target model-oriented SFT data selection method.<n>Data-DPO observes the local training feedback of the target model on different samples through one-step probing.<n>It transforms activation differences among samples into pairwise data preferences, and trains a lightweight reward model to learn target-model-aware data preferences.<n>In the final selection stage, Data-DPO further combines target model preference, external quality scores, and marginal diversity to construct a more stable and effective training subset.
  arXiv  Detail & Related papers  (2026-08-05T04:32:17Z)
• Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences [13.588231827053923]
  Direct Preference Optimization (DPO) aligns text-to-image (T2I) generation models with human preferences using pairwise preference data.<n>We propose SmPO-Diffusion, a novel method for modeling preference distributions to improve the DPO objective.<n>Our approach effectively mitigates issues of excessive optimization and objective misalignment present in existing methods.
  arXiv  Detail & Related papers  (2025-06-03T09:47:22Z)
• Towards Self-Improvement of Diffusion Models via Group Preference Optimization [10.6096255671291]
  Group Preference Optimization (GPO) is an effective self-improvement method that enhances performance without requiring external data.<n>GPO improves the accurate counting and text rendering capabilities of the Stable Diffusion 3.5 Medium by 20 percentage points.<n>As a plug-and-play method, no extra overhead is introduced during inference.
  arXiv  Detail & Related papers  (2025-05-16T10:04:57Z)
• Less is More: Improving LLM Alignment via Preference Data Selection [46.9163802899686]
  Direct Preference Optimization (DPO) has emerged as a promising approach for aligning large language models with human preferences.<n>We propose a novel margin-maximization principle for dataset curation in DPO training.<n>We show that our approach seamlessly extends to iterative DPO, yielding a roughly 3% improvement with 25% online data.
  arXiv  Detail & Related papers  (2025-02-20T13:45:17Z)
• DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization [75.55167570591063]
  We propose DreamDPO, an optimization-based framework that integrates human preferences into the 3D generation process.<n>DreamDPO reduces reliance on precise pointwise quality evaluations while enabling fine-grained controllability.<n>Experiments demonstrate that DreamDPO achieves competitive results, and provides higher-quality and more controllable 3D content.
  arXiv  Detail & Related papers  (2025-02-05T11:03:08Z)
• Calibrated Multi-Preference Optimization for Aligning Diffusion Models [90.15024547673785]
  Calibrated Preference Optimization (CaPO) is a novel method to align text-to-image (T2I) diffusion models.<n>CaPO incorporates the general preference from multiple reward models without human annotated data.<n> Experimental results show that CaPO consistently outperforms prior methods.
  arXiv  Detail & Related papers  (2025-02-04T18:59:23Z)
• Personalized Preference Fine-tuning of Diffusion Models [75.22218338096316]
  We introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences.<n>With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way.<n>Our approach achieves an average win rate of 76% over Stable Cascade, generating images that more accurately reflect specific user preferences.
  arXiv  Detail & Related papers  (2025-01-11T22:38:41Z)
• Scalable Ranked Preference Optimization for Text-to-Image Generation [76.16285931871948]
  We investigate a scalable approach for collecting large-scale and fully synthetic datasets for DPO training. The preferences for paired images are generated using a pre-trained reward function, eliminating the need for involving humans in the annotation process. We introduce RankDPO to enhance DPO-based methods using the ranking feedback.
  arXiv  Detail & Related papers  (2024-10-23T16:42:56Z)
• MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models [85.30735602813093]
  Multi-Image Augmented Direct Preference Optimization (MIA-DPO) is a visual preference alignment approach that effectively handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse multi-image training data by extending single-image data with unrelated images arranged in grid collages or pic-in-pic formats.
  arXiv  Detail & Related papers  (2024-10-23T07:56:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.