Preference-Guided Diffusion for Multi-Objective Offline Optimization

• URL: http://arxiv.org/abs/2503.17299v1
• Date: Fri, 21 Mar 2025 16:49:38 GMT
• Title: Preference-Guided Diffusion for Multi-Objective Offline Optimization
• Authors: Yashas Annadani, Syrine Belakaria, Stefano Ermon, Stefan Bauer, Barbara E Engelhardt,
• Abstract summary: We propose a preference-guided diffusion model for offline multi-objective optimization.<n>Our guidance is a preference model trained to predict the probability that one design dominates another.<n>Our results highlight the effectiveness of classifier-guided diffusion models in generating diverse and high-quality solutions.
• Score: 64.08326521234228
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline multi-objective optimization aims to identify Pareto-optimal solutions given a dataset of designs and their objective values. In this work, we propose a preference-guided diffusion model that generates Pareto-optimal designs by leveraging a classifier-based guidance mechanism. Our guidance classifier is a preference model trained to predict the probability that one design dominates another, directing the diffusion model toward optimal regions of the design space. Crucially, this preference model generalizes beyond the training distribution, enabling the discovery of Pareto-optimal solutions outside the observed dataset. We introduce a novel diversity-aware preference guidance, augmenting Pareto dominance preference with diversity criteria. This ensures that generated solutions are optimal and well-distributed across the objective space, a capability absent in prior generative methods for offline multi-objective optimization. We evaluate our approach on various continuous offline multi-objective optimization tasks and find that it consistently outperforms other inverse/generative approaches while remaining competitive with forward/surrogate-based optimization methods. Our results highlight the effectiveness of classifier-guided diffusion models in generating diverse and high-quality solutions that approximate the Pareto front well.

Related papers

• Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment [74.25832963097658]
  Multi-Objective Alignment (MOA) aims to align responses with multiple human preference objectives.<n>We find that DPO-based MOA approaches suffer from widespread preference conflicts in the data.
  arXiv  Detail & Related papers  (2025-02-20T08:27:00Z)
• Diversity By Design: Leveraging Distribution Matching for Offline Model-Based Optimization [29.303300250713804]
  We propose Diversity in Adrial Model-based Optimization (DynAMO) as a novel method to introduce design diversity as an explicit objective into any MBO problem.<n>Our key insight is to formulate diversity as a distribution matching problem where the distribution of generated designs captures the inherent diversity contained within the offline dataset.
  arXiv  Detail & Related papers  (2025-01-30T21:43:25Z)
• ParetoFlow: Guided Flows in Multi-Objective Optimization [12.358524770639136]
  In offline multi-objective optimization (MOO), we leverage an offline dataset of designs their associated labels to simultaneously multiple objectives.<n>Recent iteration mainly employ evolutionary and Bayesian optimization, with limited attention given to the generative capabilities inherent in data.<n>Our method achieves state-of-the-art performance across various tasks.
  arXiv  Detail & Related papers  (2024-12-04T21:14:18Z)
• Model Fusion through Bayesian Optimization in Language Model Fine-Tuning [16.86812534268461]
  Fine-tuning pre-trained models for downstream tasks is a widely adopted technique known for its adaptability and reliability across various domains.<n>We introduce a novel model fusion technique that optimize both the desired metric and loss through multi-objective Bayesian optimization.<n> Experiments across various downstream tasks show considerable performance improvements using our Bayesian optimization-guided method.
  arXiv  Detail & Related papers  (2024-11-11T04:36:58Z)
• Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models [54.132297393662654]
  We introduce a hybrid method that fine-tunes cutting-edge diffusion models by optimizing reward models through RL. We demonstrate the capability of our approach to outperform the best designs in offline data, leveraging the extrapolation capabilities of reward models.
  arXiv  Detail & Related papers  (2024-05-30T03:57:29Z)
• Diffusion Model for Data-Driven Black-Box Optimization [54.25693582870226]
  We focus on diffusion models, a powerful generative AI technology, and investigate their potential for black-box optimization. We study two practical types of labels: 1) noisy measurements of a real-valued reward function and 2) human preference based on pairwise comparisons. Our proposed method reformulates the design optimization problem into a conditional sampling problem, which allows us to leverage the power of diffusion models.
  arXiv  Detail & Related papers  (2024-03-20T00:41:12Z)
• Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment [103.12563033438715]
  Alignment in artificial intelligence pursues consistency between model responses and human preferences as well as values. Existing alignment techniques are mostly unidirectional, leading to suboptimal trade-offs and poor flexibility over various objectives. We introduce controllable preference optimization (CPO), which explicitly specifies preference scores for different objectives.
  arXiv  Detail & Related papers  (2024-02-29T12:12:30Z)
• Multi-Objective Bayesian Optimization with Active Preference Learning [18.066263838953223]
  We propose a Bayesian optimization (BO) approach to identifying the most preferred solution in a multi-objective optimization (MOO) problem. To minimize the interaction cost with the decision maker (DM), we also propose an active learning strategy for the preference estimation.
  arXiv  Detail & Related papers  (2023-11-22T15:24:36Z)
• Towards Optimization and Model Selection for Domain Generalization: A Mixup-guided Solution [43.292274574847234]
  We propose Mixup guided optimization and selection techniques for domain generalization. For optimization, we utilize an out-of-distribution dataset that can guide the preference direction. For model selection, we generate a validation dataset with a closer distance to the target distribution.
  arXiv  Detail & Related papers  (2022-09-01T02:18:00Z)
• Optimization-Inspired Learning with Architecture Augmentations and Control Mechanisms for Low-Level Vision [74.9260745577362]
  This paper proposes a unified optimization-inspired learning framework to aggregate Generative, Discriminative, and Corrective (GDC) principles. We construct three propagative modules to effectively solve the optimization models with flexible combinations. Experiments across varied low-level vision tasks validate the efficacy and adaptability of GDC.
  arXiv  Detail & Related papers  (2020-12-10T03:24:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.