Preference optimization of protein language models as a multi-objective binder design paradigm

• URL: http://arxiv.org/abs/2403.04187v1
• Date: Thu, 7 Mar 2024 03:36:03 GMT
• Title: Preference optimization of protein language models as a multi-objective binder design paradigm
• Authors: Pouria Mistani, Venkatesh Mysore
• Abstract summary: We present a multi-objective binder design paradigm based on instruction fine-tuning and direct preference optimization. We show the proposed alignment strategy enables ProtGPT2 to effectively design binders conditioned on specified receptors and a drug developability criterion.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present a multi-objective binder design paradigm based on instruction fine-tuning and direct preference optimization (DPO) of autoregressive protein language models (pLMs). Multiple design objectives are encoded in the language model through direct optimization on expert curated preference sequence datasets comprising preferred and dispreferred distributions. We show the proposed alignment strategy enables ProtGPT2 to effectively design binders conditioned on specified receptors and a drug developability criterion. Generated binder samples demonstrate median isoelectric point (pI) improvements by $17\%-60\%$.

Related papers

• Preference-Guided Diffusion for Multi-Objective Offline Optimization [64.08326521234228]
  We propose a preference-guided diffusion model for offline multi-objective optimization. Our guidance is a preference model trained to predict the probability that one design dominates another. Our results highlight the effectiveness of classifier-guided diffusion models in generating diverse and high-quality solutions.
  arXiv  Detail & Related papers  (2025-03-21T16:49:38Z)
• Robust Multi-Objective Preference Alignment with Online DPO [6.434799451791957]
  Multi-objective preference alignment is critical for developing AI systems that are personalizable, helpful, and safe. Existing approaches are either computationally expensive to train or do not sufficiently steer model behaviors. This paper introduces the Multi-Objective Online DPO algorithm, designed to robustly and efficiently align model behaviors with multiple, potentially conflicting human preferences.
  arXiv  Detail & Related papers  (2025-03-01T02:01:49Z)
• Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment [74.25832963097658]
  Multi-Objective Alignment (MOA) aims to align responses with multiple human preference objectives. We find that DPO-based MOA approaches suffer from widespread preference conflicts in the data.
  arXiv  Detail & Related papers  (2025-02-20T08:27:00Z)
• Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment [45.45508377432791]
  This paper introduces Reward-Aware Preference Optimization (RPO), a mathematical framework that unifies popular preference optimization techniques. RPO provides a structured approach to disentangle and systematically study the impact of various design choices. We propose a new experimental setup that enables the clean and direct ablation of such design choices.
  arXiv  Detail & Related papers  (2025-01-31T22:39:04Z)
• Diversity By Design: Leveraging Distribution Matching for Offline Model-Based Optimization [29.303300250713804]
  We propose Diversity in Adrial Model-based Optimization (DynAMO) as a novel method to introduce design diversity as an explicit objective into any MBO problem. Our key insight is to formulate diversity as a distribution matching problem where the distribution of generated designs captures the inherent diversity contained within the offline dataset.
  arXiv  Detail & Related papers  (2025-01-30T21:43:25Z)
• Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization [14.50339880957898]
  We aim to improve the preference optimization pipeline by taking a closer look at preference data generation and training regularization techniques. For preference data generation, we propose an iterative pairwise ranking mechanism that derives preference ranking of completions using pairwise comparison signals. For training regularization, we observe that preference optimization tends to achieve better convergence when the LLM predicted likelihood of preferred samples gets slightly reduced.
  arXiv  Detail & Related papers  (2024-11-07T23:03:11Z)
• Preference Optimization with Multi-Sample Comparisons [53.02717574375549]
  We introduce a novel approach that extends post-training to include multi-sample comparisons. These approaches fail to capture critical characteristics such as generative diversity and bias. We demonstrate that multi-sample comparison is more effective in optimizing collective characteristics than single-sample comparison.
  arXiv  Detail & Related papers  (2024-10-16T00:59:19Z)
• Preference Alignment Improves Language Model-Based TTS [76.70693823683091]
  preference alignment algorithms adjust LMs to align with the preferences of reward models, enhancing the desirability of the generated content. With a 1.15B parameter LM-based TTS model, we demonstrate that preference alignment consistently improves intelligibility, speaker similarity, and proxy subjective evaluation scores.
  arXiv  Detail & Related papers  (2024-09-19T01:58:19Z)
• mDPO: Conditional Preference Optimization for Multimodal Large Language Models [52.607764280030196]
  Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment. Recent works have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement. We propose mDPO, a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference.
  arXiv  Detail & Related papers  (2024-06-17T17:59:58Z)
• Hybrid Preference Optimization: Augmenting Direct Preference Optimization with Auxiliary Objectives [0.5120567378386615]
  We propose a hybrid approach to aligning large language models (LLMs) With a simple augmentation to the implicit reward decomposition of DPO, we allow for tuning LLMs to maximize a set of arbitrary auxiliary rewards. The proposed method, Hybrid Preference Optimization (HPO), shows the ability to effectively generalize to both user preferences and auxiliary designer objectives.
  arXiv  Detail & Related papers  (2024-05-28T08:35:48Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)
• Annotation-Efficient Preference Optimization for Language Model Alignment [3.726173629675064]
  We show how to use the limited annotation budget to create an effective preference dataset. We evaluate the performance of Direct Preference Optimization (DPO) using AEPO and show that it outperforms models trained using a standard DPO with the same annotation budget.
  arXiv  Detail & Related papers  (2024-05-22T11:23:03Z)
• Diffusion Model for Data-Driven Black-Box Optimization [54.25693582870226]
  We focus on diffusion models, a powerful generative AI technology, and investigate their potential for black-box optimization. We study two practical types of labels: 1) noisy measurements of a real-valued reward function and 2) human preference based on pairwise comparisons. Our proposed method reformulates the design optimization problem into a conditional sampling problem, which allows us to leverage the power of diffusion models.
  arXiv  Detail & Related papers  (2024-03-20T00:41:12Z)
• Functional Graphical Models: Structure Enables Offline Data-Driven Optimization [111.28605744661638]
  We show how structure can enable sample-efficient data-driven optimization. We also present a data-driven optimization algorithm that infers the FGM structure itself.
  arXiv  Detail & Related papers  (2024-01-08T22:33:14Z)
• Towards Optimization and Model Selection for Domain Generalization: A Mixup-guided Solution [43.292274574847234]
  We propose Mixup guided optimization and selection techniques for domain generalization. For optimization, we utilize an out-of-distribution dataset that can guide the preference direction. For model selection, we generate a validation dataset with a closer distance to the target distribution.
  arXiv  Detail & Related papers  (2022-09-01T02:18:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.