REINFORCE-ING Chemical Language Models for Drug Discovery

• URL: http://arxiv.org/abs/2501.15971v2
• Date: Tue, 04 Nov 2025 19:38:48 GMT
• Title: REINFORCE-ING Chemical Language Models for Drug Discovery
• Authors: Morgan Thomas, Albert Bou, Jose Carlos Gómez-Tamayo, Gary Tresadern, Mazen Ahmad, Gianni De Fabritiis,
• Abstract summary: reinforcement learning can efficiently traverse large chemical spaces for drug discovery.<n>We investigate the effect of different components from RL theory including experience replay, hill-climbing, baselines to reduce variance.<n>We apply our learnings to practical drug discovery by demonstrating enhanced learning efficiency on frontier binding affinity models.
• Score: 4.361479497880884
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Chemical language models, combined with reinforcement learning (RL), have shown significant promise to efficiently traverse large chemical spaces for drug discovery. However, the performance of various RL algorithms and their best practices for practical drug discovery are still unclear. Here, starting from the principles of the REINFORCE algorithm, we investigate the effect of different components from RL theory including experience replay, hill-climbing, baselines to reduce variance, and alternative reward shaping. We propose a new regularization method more aligned to REINFORCE than current standard practices, and demonstrate how RL hyperparameters can be fine-tuned for effectiveness and efficiency. Lastly, we apply our learnings to practical drug discovery by demonstrating enhanced learning efficiency on frontier binding affinity models by using Boltz2 as a reward model. We share our RL models used in the ACEGEN repository, and hope the experiments here act as a guide to researchers applying RL to chemical language models for drug discovery.

Related papers

• Enhancing Diffusion-based Restoration Models via Difficulty-Adaptive Reinforcement Learning with IQA Reward [93.04811239892852]
  Reinforcement Learning (RL) has recently been incorporated into diffusion models.<n>In this paper, we investigate how to effectively integrate RL into diffusion-based restoration models.
  arXiv  Detail & Related papers  (2025-11-03T14:57:57Z)
• Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts [113.0656076371565]
  We propose a novel router-aware approach to optimize importance sampling weights in off-policy reinforcement learning (RL)<n>Specifically, we design a rescaling strategy guided by router logits, which effectively reduces gradient variance and mitigates training divergence.<n> Experimental results demonstrate that our method significantly improves both the convergence stability and the final performance of MoE models.
  arXiv  Detail & Related papers  (2025-10-27T05:47:48Z)
• The Art of Scaling Reinforcement Learning Compute for LLMs [52.71086085139566]
  Reinforcement learning (RL) has become central to training large language models.<n>Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute.<n>We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours.
  arXiv  Detail & Related papers  (2025-10-15T17:43:03Z)
• Leveraging Genetic Algorithms for Efficient Demonstration Generation in Real-World Reinforcement Learning Environments [0.8602553195689513]
  Reinforcement Learning (RL) has demonstrated significant potential in certain real-world industrial applications.<n>This study investigates the utilization of Genetic Algorithms (GAs) as a mechanism for improving RL performance.<n>We propose a novel approach in which GA-generated expert demonstrations are used to enhance policy learning.
  arXiv  Detail & Related papers  (2025-07-01T14:04:17Z)
• Reparameterized LLM Training via Orthogonal Equivalence Transformation [54.80172809738605]
  We present POET, a novel training algorithm that uses Orthogonal Equivalence Transformation to optimize neurons.<n>POET can stably optimize the objective function with improved generalization.<n>We develop efficient approximations that make POET flexible and scalable for training large-scale neural networks.
  arXiv  Detail & Related papers  (2025-06-09T17:59:34Z)
• The Hallucination Dilemma: Factuality-Aware Reinforcement Learning for Large Reasoning Models [63.98194996746229]
  Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization.<n>However, reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations.<n>We propose Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification.
  arXiv  Detail & Related papers  (2025-05-30T14:23:32Z)
• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning [50.02117478165099]
  We show that large-scale reinforcement learning can significantly enhance the reasoning capabilities of strong, small- and mid-sized models.<n>We propose a simple yet effective approach: first training on math-only prompts, then on code-only prompts.
  arXiv  Detail & Related papers  (2025-05-22T08:50:47Z)
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining [74.83412846804977]
  Reinforcement learning (RL)-based fine-tuning has become a crucial step in post-training language models. We present a systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch.
  arXiv  Detail & Related papers  (2025-04-10T17:15:53Z)
• De Novo Molecular Design Enabled by Direct Preference Optimization and Curriculum Learning [0.0]
  De novo molecular design has extensive applications in drug discovery and materials science. The vast chemical space renders direct molecular searches computationally prohibitive, while traditional experimental screening is both time- and labor-intensive. Direct Preference Optimization (DPO) from NLP uses molecular score-based sample pairs to maximize the likelihood difference between high- and low-quality molecules.
  arXiv  Detail & Related papers  (2025-04-02T06:00:21Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE) RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers. Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimization [53.27954325490941]
  Finetuning a Large Language Model (LLM) is crucial for generating results towards specific objectives. This research introduces a novel reinforcement learning algorithm to finetune a drug optimization LLM-based generative model.
  arXiv  Detail & Related papers  (2025-02-11T04:00:21Z)
• Test-Time Training Scaling for Chemical Exploration in Drug Design [3.1406146587437904]
  We propose a new benchmark to discover dissimilar molecules that possess similar bioactivity. We show that a population of RL agents can solve the benchmark, while a single agent cannot. We also find that cooperative strategies are not significantly better than independent agents.
  arXiv  Detail & Related papers  (2025-01-31T14:11:10Z)
• Numerical Pruning for Efficient Autoregressive Models [87.56342118369123]
  This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning.<n>Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and modules, respectively.<n>To verify the effectiveness of our method, we provide both theoretical support and extensive experiments.
  arXiv  Detail & Related papers  (2024-12-17T01:09:23Z)
• Pathway-Guided Optimization of Deep Generative Molecular Design Models for Cancer Therapy [1.8210200978176423]
  The junction tree variational autoencoder (JTVAE) has been shown to be an efficient generative model. We show how a pharmacodynamic model, assessing the therapeutic efficacy of a drug-like small molecule, can be incorporated for effective latent space optimization.
  arXiv  Detail & Related papers  (2024-11-05T19:20:30Z)
• BatGPT-Chem: A Foundation Large Model For Retrosynthesis Prediction [65.93303145891628]
  BatGPT-Chem is a large language model with 15 billion parameters, tailored for enhanced retrosynthesis prediction. Our model captures a broad spectrum of chemical knowledge, enabling precise prediction of reaction conditions. This development empowers chemists to adeptly address novel compounds, potentially expediting the innovation cycle in drug manufacturing and materials science.
  arXiv  Detail & Related papers  (2024-08-19T05:17:40Z)
• Text-Augmented Multimodal LLMs for Chemical Reaction Condition Recommendation [50.639325453203504]
  MM-RCR is a text-augmented multimodal LLM that learns a unified reaction representation from SMILES, reaction graphs, and textual corpus for chemical reaction recommendation (RCR) Our results demonstrate that MM-RCR achieves state-of-the-art performance on two open benchmark datasets.
  arXiv  Detail & Related papers  (2024-07-21T12:27:26Z)
• Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review [63.31328039424469]
  This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning.
  arXiv  Detail & Related papers  (2024-07-18T17:35:32Z)
• ACEGEN: Reinforcement learning of generative chemical agents for drug discovery [4.966722586536789]
  ACEGEN is a comprehensive and streamlined toolkit for generative drug design. TorchRL is a modern RL library that offers thoroughly tested reusable components. We show examples of ACEGEN applied in multiple drug discovery case studies.
  arXiv  Detail & Related papers  (2024-05-07T20:30:14Z)
• Contextual Molecule Representation Learning from Chemical Reaction Knowledge [24.501564702095937]
  We introduce REMO, a self-supervised learning framework that takes advantage of well-defined atom-combination rules in common chemistry. REMO pre-trains graph/Transformer encoders on 1.7 million known chemical reactions in the literature.
  arXiv  Detail & Related papers  (2024-02-21T12:58:40Z)
• ChemGymRL: An Interactive Framework for Reinforcement Learning for Digital Chemistry [2.350237106287331]
  This paper provides a simulated laboratory for making use of Reinforcement Learning (RL) for chemical discovery. Since RL is fairly data intensive, training agents on-the-fly' by taking actions in the real world is infeasible and possibly dangerous. We introduce a set of highly customizable and open-source RL environments, ChemGymRL, based on the standard Open AI Gym template.
  arXiv  Detail & Related papers  (2023-05-23T15:56:17Z)
• Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation [34.26748101294543]
  We propose a novel framework that generates pharmacochemically acceptable molecules with large docking scores. Our method constrains the generated molecules to a realistic and qualified chemical space and effectively explores the space to find drugs. Our model produces molecules of higher quality compared to existing methods while achieving state-of-the-art performance on two of three targets.
  arXiv  Detail & Related papers  (2021-10-04T07:21:00Z)
• Learning to Prune Deep Neural Networks via Reinforcement Learning [64.85939668308966]
  PuRL is a deep reinforcement learning based algorithm for pruning neural networks. It achieves sparsity and accuracy comparable to current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-09T13:06:07Z)
• Deep Learning for Virtual Screening: Five Reasons to Use ROC Cost Functions [80.12620331438052]
  deep learning has become an important tool for rapid screening of billions of molecules in silico for potential hits containing desired chemical features. Despite its importance, substantial challenges persist in training these models, such as severe class imbalance, high decision thresholds, and lack of ground truth labels in some datasets. We argue in favor of directly optimizing the receiver operating characteristic (ROC) in such cases, due to its robustness to class imbalance.
  arXiv  Detail & Related papers  (2020-06-25T08:46:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.