Preference optimization of protein language models as a multi-objective
binder design paradigm
- URL: http://arxiv.org/abs/2403.04187v1
- Date: Thu, 7 Mar 2024 03:36:03 GMT
- Title: Preference optimization of protein language models as a multi-objective
binder design paradigm
- Authors: Pouria Mistani, Venkatesh Mysore
- Abstract summary: We present a multi-objective binder design paradigm based on instruction fine-tuning and direct preference optimization.
We show the proposed alignment strategy enables ProtGPT2 to effectively design binders conditioned on specified receptors and a drug developability criterion.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: We present a multi-objective binder design paradigm based on instruction
fine-tuning and direct preference optimization (DPO) of autoregressive protein
language models (pLMs). Multiple design objectives are encoded in the language
model through direct optimization on expert curated preference sequence
datasets comprising preferred and dispreferred distributions. We show the
proposed alignment strategy enables ProtGPT2 to effectively design binders
conditioned on specified receptors and a drug developability criterion.
Generated binder samples demonstrate median isoelectric point (pI) improvements
by $17\%-60\%$.
Related papers
- mDPO: Conditional Preference Optimization for Multimodal Large Language Models [52.607764280030196]
Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment.
Recent works have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement.
We propose mDPO, a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference.
arXiv Detail & Related papers (2024-06-17T17:59:58Z) - Hybrid Preference Optimization: Augmenting Direct Preference Optimization with Auxiliary Objectives [0.5120567378386615]
We propose a hybrid approach to aligning large language models (LLMs)
With a simple augmentation to the implicit reward decomposition of DPO, we allow for tuning LLMs to maximize a set of arbitrary auxiliary rewards.
The proposed method, Hybrid Preference Optimization (HPO), shows the ability to effectively generalize to both user preferences and auxiliary designer objectives.
arXiv Detail & Related papers (2024-05-28T08:35:48Z) - Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
We introduce a novel closed-form formulation for direct preference optimization using multiple reference models.
The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models.
Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
arXiv Detail & Related papers (2024-05-26T00:29:04Z) - Annotation-Efficient Preference Optimization for Language Model Alignment [3.726173629675064]
We show how to use the limited annotation budget to create an effective preference dataset.
We evaluate the performance of Direct Preference Optimization (DPO) using AEPO and show that it outperforms models trained using a standard DPO with the same annotation budget.
arXiv Detail & Related papers (2024-05-22T11:23:03Z) - Diffusion Model for Data-Driven Black-Box Optimization [54.25693582870226]
We focus on diffusion models, a powerful generative AI technology, and investigate their potential for black-box optimization.
We study two practical types of labels: 1) noisy measurements of a real-valued reward function and 2) human preference based on pairwise comparisons.
Our proposed method reformulates the design optimization problem into a conditional sampling problem, which allows us to leverage the power of diffusion models.
arXiv Detail & Related papers (2024-03-20T00:41:12Z) - FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema [36.65009632307124]
We propose Free-from Instruction-oriented Prompt Optimization (FIPO) to improve task performance of large language models (LLMs)
FIPO uses a modular APO template that dynamically integrate the naive task instruction, optional instruction responses, and optional ground truth to produce finely optimized prompts.
We validate FIPO framework across five public benchmarks and three testing models.
arXiv Detail & Related papers (2024-02-19T03:56:44Z) - Functional Graphical Models: Structure Enables Offline Data-Driven
Optimization [121.57202302457135]
We show how structure can enable sample-efficient data-driven optimization.
We also present a data-driven optimization algorithm that infers the FGM structure itself.
arXiv Detail & Related papers (2024-01-08T22:33:14Z) - Protein Design with Guided Discrete Diffusion [67.06148688398677]
A popular approach to protein design is to combine a generative model with a discriminative model for conditional sampling.
We propose diffusioN Optimized Sampling (NOS), a guidance method for discrete diffusion models.
NOS makes it possible to perform design directly in sequence space, circumventing significant limitations of structure-based methods.
arXiv Detail & Related papers (2023-05-31T16:31:24Z) - Towards Optimization and Model Selection for Domain Generalization: A
Mixup-guided Solution [43.292274574847234]
We propose Mixup guided optimization and selection techniques for domain generalization.
For optimization, we utilize an out-of-distribution dataset that can guide the preference direction.
For model selection, we generate a validation dataset with a closer distance to the target distribution.
arXiv Detail & Related papers (2022-09-01T02:18:00Z) - SimPO: Simultaneous Prediction and Optimization [3.181417685380586]
We propose a formulation for the Simultaneous Prediction and Optimization (SimPO) framework.
This framework introduces the use of a joint weighted loss of a decision-driven predictive ML model and an optimization objective function.
arXiv Detail & Related papers (2022-03-31T20:01:36Z) - Optimization-Inspired Learning with Architecture Augmentations and
Control Mechanisms for Low-Level Vision [74.9260745577362]
This paper proposes a unified optimization-inspired learning framework to aggregate Generative, Discriminative, and Corrective (GDC) principles.
We construct three propagative modules to effectively solve the optimization models with flexible combinations.
Experiments across varied low-level vision tasks validate the efficacy and adaptability of GDC.
arXiv Detail & Related papers (2020-12-10T03:24:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.