BalancedDPO: Adaptive Multi-Metric Alignment

• URL: http://arxiv.org/abs/2503.12575v1
• Date: Sun, 16 Mar 2025 17:06:00 GMT
• Title: BalancedDPO: Adaptive Multi-Metric Alignment
• Authors: Dipesh Tamboli, Souradip Chakraborty, Aditya Malusare, Biplab Banerjee, Amrit Singh Bedi, Vaneet Aggarwal,
• Abstract summary: We present BalancedDPO, a novel extension of Direct Preference Optimization (DPO)<n>We simultaneously align T2I diffusion models with multiple metrics, including human preference, CLIP score, and aesthetic quality.<n>Our evaluations show that BalancedDPO achieves state-of-the-art results, outperforming existing approaches across all major metrics.
• Score: 44.051100004689296
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-image (T2I) diffusion models have made remarkable advancements, yet aligning them with diverse preferences remains a persistent challenge. Current methods often optimize single metrics or depend on narrowly curated datasets, leading to overfitting and limited generalization across key visual quality metrics. We present BalancedDPO, a novel extension of Direct Preference Optimization (DPO) that addresses these limitations by simultaneously aligning T2I diffusion models with multiple metrics, including human preference, CLIP score, and aesthetic quality. Our key novelty lies in aggregating consensus labels from diverse metrics in the preference distribution space as compared to existing reward mixing approaches, enabling robust and scalable multi-metric alignment while maintaining the simplicity of the standard DPO pipeline that we refer to as BalancedDPO. Our evaluations on the Pick-a-Pic, PartiPrompt and HPD datasets show that BalancedDPO achieves state-of-the-art results, outperforming existing approaches across all major metrics. BalancedDPO improves the average win rates by 15%, 7.1%, and 10.3% on Pick-a-pic, PartiPrompt and HPD, respectively, from the DiffusionDPO.

Related papers

• Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO [51.22869332661607]
  We decompose the performance gap between reinforcement learning from human feedback and direct preference optimization under a representation gap.<n>We show that RLHF, DPO, or online DPO can outperform one another depending on the type of model mis-specifications.
  arXiv  Detail & Related papers  (2025-05-26T09:54:02Z)
• Towards Self-Improvement of Diffusion Models via Group Preference Optimization [10.6096255671291]
  Group Preference Optimization (GPO) is an effective self-improvement method that enhances performance without requiring external data.<n>GPO improves the accurate counting and text rendering capabilities of the Stable Diffusion 3.5 Medium by 20 percentage points.<n>As a plug-and-play method, no extra overhead is introduced during inference.
  arXiv  Detail & Related papers  (2025-05-16T10:04:57Z)
• Less is More: Improving LLM Alignment via Preference Data Selection [46.9163802899686]
  Direct Preference Optimization (DPO) has emerged as a promising approach for aligning large language models with human preferences. We propose a novel margin-maximization principle for dataset curation in DPO training. By using just 10% of the Ultrafeedback dataset, our approach achieves 3% to 8% improvements across various Llama and Mistral series models.
  arXiv  Detail & Related papers  (2025-02-20T13:45:17Z)
• Dual Caption Preference Optimization for Diffusion Models [51.223275938663235]
  We propose Dual Caption Preference Optimization (DCPO), a novel approach that utilizes two distinct captions to mitigate irrelevant prompts. Our experiments show that DCPO significantly improves image quality and relevance to prompts, outperforming Stable Diffusion (SD) 2.1, SFT_Chosen, Diffusion-DPO, and MaPO across multiple metrics.
  arXiv  Detail & Related papers  (2025-02-09T20:34:43Z)
• Calibrated Multi-Preference Optimization for Aligning Diffusion Models [92.90660301195396]
  Calibrated Preference Optimization (CaPO) is a novel method to align text-to-image (T2I) diffusion models.<n>CaPO incorporates the general preference from multiple reward models without human annotated data.<n> Experimental results show that CaPO consistently outperforms prior methods.
  arXiv  Detail & Related papers  (2025-02-04T18:59:23Z)
• SWEPO: Simultaneous Weighted Preference Optimization for Group Contrastive Alignment [16.230186347702737]
  We propose Simultaneous Weighted Preference Optimization (SWEPO) SWEPO incorporates multiple responses per query and prioritizes those that deviate most from the average reward. We prove that such multi-preference sampling lowers alignment bias, bounding the expected deviation from the true acceptable-response distribution at a rate of $mathcalO(tfrac1sqrtk)$.
  arXiv  Detail & Related papers  (2024-12-05T21:50:22Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights [73.9088920210495]
  We propose a token-level importance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward. TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks.
  arXiv  Detail & Related papers  (2024-10-06T04:03:00Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence [31.03305638930844]
  Direct Preference Optimization (DPO) has emerged as a prominent algorithm for the direct and robust alignment of Large Language Models with human preferences.<n>Despite its promising efficacy, DPO faces a notable drawback: "verbosity"<n>We propose that the issue also stems from an inherent algorithmic length reliance in DPO.
  arXiv  Detail & Related papers  (2024-06-16T14:24:30Z)
• Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization [34.29965046863887]
  Triple Preference Optimization (TPO) is a new preference learning method designed to enhance both reasoning and instruction-following abilities.<n>TPO achieves significant improvements over existing methods without substantially increasing response length across different dataset sizes.
  arXiv  Detail & Related papers  (2024-05-26T20:18:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.