Protein Discovery with Discrete Walk-Jump Sampling

• URL: http://arxiv.org/abs/2306.12360v2
• Date: Fri, 15 Mar 2024 19:16:01 GMT
• Title: Protein Discovery with Discrete Walk-Jump Sampling
• Authors: Nathan C. Frey, Daniel Berenberg, Karina Zadorozhny, Joseph Kleinhenz, Julien Lafrance-Vanasse, Isidro Hotzel, Yan Wu, Stephen Ra, Richard Bonneau, Kyunghyun Cho, Andreas Loukas, Vladimir Gligorijevic, Saeed Saremi,
• Abstract summary: We learn a smoothed energy function, sampling from the smoothed data manifold with Langevin Markov chain Monte Carlo, and projecting back to the true data manifold with one-step denoising. Our Discrete Walk-Jump Sampling formalism combines the contrastive divergence training of an energy-based model and improved sample quality of a score-based model. We evaluate the robustness of our approach on generative modeling of antibody proteins and introduce the distributional conformity score to benchmark protein generative models.
• Score: 41.01079393600248
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We resolve difficulties in training and sampling from a discrete generative model by learning a smoothed energy function, sampling from the smoothed data manifold with Langevin Markov chain Monte Carlo (MCMC), and projecting back to the true data manifold with one-step denoising. Our Discrete Walk-Jump Sampling formalism combines the contrastive divergence training of an energy-based model and improved sample quality of a score-based model, while simplifying training and sampling by requiring only a single noise level. We evaluate the robustness of our approach on generative modeling of antibody proteins and introduce the distributional conformity score to benchmark protein generative models. By optimizing and sampling from our models for the proposed distributional conformity score, 97-100% of generated samples are successfully expressed and purified and 70% of functional designs show equal or improved binding affinity compared to known functional antibodies on the first attempt in a single round of laboratory experiments. We also report the first demonstration of long-run fast-mixing MCMC chains where diverse antibody protein classes are visited in a single MCMC chain.

Related papers

• Reinforced sequential Monte Carlo for amortised sampling [49.92678178064033]
  We state a connection between sequential Monte Carlo (SMC) and neural sequential samplers trained by maximum-entropy reinforcement learning (MaxEnt RL)<n>We describe techniques for stable joint training of proposals and twist functions and an adaptive weight tempering scheme to reduce training signal variance.
  arXiv  Detail & Related papers  (2025-10-13T17:59:11Z)
• Amortized Sampling with Transferable Normalizing Flows [65.48838168417564]
  Prose is a transferable normalizing flow trained on a corpus of peptide molecular dynamics trajectories up to 8 residues in length.<n>We show that Prose is a proposal for a variety of sampling algorithms, finding a simple importance sampling-based finetuning procedure to achieve superior performance.<n>We open-source the Prose dataset to further stimulate research into amortized sampling methods and finetuning objectives.
  arXiv  Detail & Related papers  (2025-08-25T16:28:18Z)
• Noise Conditional Variational Score Distillation [60.38982038894823]
  Noise Conditional Variational Score Distillation (NCVSD) is a novel method for distilling pretrained diffusion models into generative denoisers.<n>By integrating this insight into the Variational Score Distillation framework, we enable scalable learning of generative denoisers.
  arXiv  Detail & Related papers  (2025-06-11T06:01:39Z)
• Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo [32.46884330460211]
  We propose a simple yet effective approach to enhance sample diversity in Gradient Markov Chain Monte Carlo. This approach produces a more diverse set of samples, allowing faster mixing within the same computational budget. Our experiments on image classification tasks, including OOD robustness, diversity, loss surface analyses, and a comparative study with Hamiltonian Monte Carlo, demonstrate the superiority of the proposed approach.
  arXiv  Detail & Related papers  (2025-03-02T02:42:50Z)
• CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation [9.12693573953231]
  We first observe an interesting phenomenon: the relationship between the change of generation outputs and the scale of initial noise perturbation is highly linear through the diffusion ODE sampling. We propose a novel Controllable and Constrained Sampling method (CCS) together with a new controller algorithm for diffusion models to sample with desired statistical properties. Results show that our CCS method achieves more precisely controlled sampling while maintaining superior sample quality and diversity.
  arXiv  Detail & Related papers  (2025-02-07T05:30:48Z)
• The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications [1.8416014644193066]
  We compare learned neural scores to the scores of two kinds of analytically tractable distributions. We claim that the learned neural score is dominated by its linear (Gaussian) approximation for moderate to high noise scales. We show that this allows the skipping of the first 15-30% of sampling steps while maintaining high sample quality.
  arXiv  Detail & Related papers  (2024-12-12T21:31:27Z)
• Provable Statistical Rates for Consistency Diffusion Models [87.28777947976573]
  Despite the state-of-the-art performance, diffusion models are known for their slow sample generation due to the extensive number of steps involved. This paper contributes towards the first statistical theory for consistency models, formulating their training as a distribution discrepancy minimization problem.
  arXiv  Detail & Related papers  (2024-06-23T20:34:18Z)
• Boosting Diffusion Models with Moving Average Sampling in Frequency Domain [101.43824674873508]
  Diffusion models rely on the current sample to denoise the next one, possibly resulting in denoising instability. In this paper, we reinterpret the iterative denoising process as model optimization and leverage a moving average mechanism to ensemble all the prior samples. We name the complete approach "Moving Average Sampling in Frequency domain (MASF)"
  arXiv  Detail & Related papers  (2024-03-26T16:57:55Z)
• Iterated Denoising Energy Matching for Sampling from Boltzmann Densities [109.23137009609519]
  Iterated Denoising Energy Matching (iDEM) iDEM alternates between (I) sampling regions of high model density from a diffusion-based sampler and (II) using these samples in our matching objective. We show that the proposed approach achieves state-of-the-art performance on all metrics and trains $2-5times$ faster.
  arXiv  Detail & Related papers  (2024-02-09T01:11:23Z)
• MCMC-Correction of Score-Based Diffusion Models for Model Composition [2.682859657520006]
  Diffusion models can be parameterized in terms of a score or an energy function.<n>We introduce a novel MH-like acceptance rule based on line integration of the score function.
  arXiv  Detail & Related papers  (2023-07-26T07:50:41Z)
• Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning [85.55727213502402]
  We focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks. We propose Sample-specific Ensemble of Source Models (SESoM) SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs.
  arXiv  Detail & Related papers  (2022-10-23T01:33:16Z)
• Selectively increasing the diversity of GAN-generated samples [8.980453507536017]
  We propose a novel method to selectively increase the diversity of GAN-generated samples. We show the superiority of our method in a synthetic benchmark as well as a real-life scenario simulating data from the Zero Degree Calorimeter of ALICE experiment in CERN.
  arXiv  Detail & Related papers  (2022-07-04T16:27:06Z)
• Reconstructing the Universe with Variational self-Boosted Sampling [7.922637707393503]
  Traditional algorithms such as Hamiltonian Monte Carlo (HMC) are computationally inefficient due to generating correlated samples. Here we develop a hybrid scheme called variational self-boosted sampling (VBS) to mitigate the drawbacks of both algorithms. VBS generates better quality of samples than simple VI approaches and reduces the correlation length in the sampling phase by a factor of 10-50 over using only HMC.
  arXiv  Detail & Related papers  (2022-06-28T21:30:32Z)
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling [0.6554326244334867]
  We show that sampling the equilibrium distribution via Markov Chain Monte Carlo can be dramatically accelerated using biased sampling techniques. We also show that this sampling technique can be exploited to improve the computation of the log-likelihood gradient during the training too.
  arXiv  Detail & Related papers  (2022-06-02T21:29:01Z)
• Robust Finite Mixture Regression for Heterogeneous Targets [70.19798470463378]
  We propose an FMR model that finds sample clusters and jointly models multiple incomplete mixed-type targets simultaneously. We provide non-asymptotic oracle performance bounds for our model under a high-dimensional learning framework. The results show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-12T03:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.