GeLLMO: Generalizing Large Language Models for Multi-property Molecule Optimization

• URL: http://arxiv.org/abs/2502.13398v2
• Date: Tue, 27 May 2025 17:37:58 GMT
• Title: GeLLMO: Generalizing Large Language Models for Multi-property Molecule Optimization
• Authors: Vishal Dey, Xiao Hu, Xia Ning,
• Abstract summary: Large Language Models (LLMs) demonstrate remarkable out-of-domain generalizability to novel tasks.<n>We introduce MuMOInstruct, the first high-quality instruction-tuning dataset specifically focused on complex multi-property molecule optimization tasks.<n>We develop GeLLMOs, a series of instruction-tuned LLMs for molecule optimization.
• Score: 2.152507712409726
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite recent advancements, most computational methods for molecule optimization are constrained to single- or double-property optimization tasks and suffer from poor scalability and generalizability to novel optimization tasks. Meanwhile, Large Language Models (LLMs) demonstrate remarkable out-of-domain generalizability to novel tasks. To demonstrate LLMs' potential for molecule optimization, we introduce MuMOInstruct, the first high-quality instruction-tuning dataset specifically focused on complex multi-property molecule optimization tasks. Leveraging MuMOInstruct, we develop GeLLMOs, a series of instruction-tuned LLMs for molecule optimization. Extensive evaluations across 5 in-domain and 5 out-of-domain tasks demonstrate that GeLLMOs consistently outperform state-of-the-art baselines. GeLLMOs also exhibit outstanding zero-shot generalization to unseen tasks, significantly outperforming powerful closed-source LLMs. Such strong generalizability demonstrates the tremendous potential of GeLLMOs as foundational models for molecule optimization, thereby tackling novel optimization tasks without resource-intensive retraining. MuMOInstruct, models, and code are accessible through https://github.com/ninglab/GeLLMO.

Related papers

• Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs [9.085280547983091]
  DeBoP is an automatic optimization method, which focuses on the optimization directly on the behavior of LwLLMs.<n>We evaluate DeBoP on seven challenging tasks where state-of-the-art LLMs excel but LwLLMs generally underperform.<n>DeBoP-optimized LwLLMs surpass GPT-3.5 on most tasks while reducing computational time by approximately 60%.
  arXiv  Detail & Related papers  (2025-06-06T02:40:42Z)
• Large Language Models for Controllable Multi-property Multi-objective Molecule Optimization [2.152507712409726]
  We introduce C-MuMOInstruct, the first instruction-tuning dataset focused on multi-property optimization with explicit, property-specific objectives.<n>We develop GeLLMO-Cs, a series of instruction-tuned LLMs that can perform targeted property-specific optimization.<n>Our experiments show that GeLLMO-Cs consistently outperform strong baselines, achieving up to 126% higher success rate.
  arXiv  Detail & Related papers  (2025-05-29T20:29:14Z)
• GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers [52.17222304851524]
  We introduce GReaTer, a novel prompt optimization technique that directly incorporates gradient information over task-specific reasoning.<n>By utilizing task loss gradients, GReaTer enables self-optimization of prompts for open-source, lightweight language models.<n> GReaTer consistently outperforms previous state-of-the-art prompt optimization methods.
  arXiv  Detail & Related papers  (2024-12-12T20:59:43Z)
• Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization [65.64108848398696]
  We introduce a preference optimization (PO) process to enhance the multimodal reasoning capabilities of MLLMs.<n>Specifically, we design an automated preference data construction pipeline to create MMPR, a high-quality, large-scale multimodal reasoning preference dataset.<n>We explore integrating PO with MLLMs, developing a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance.
  arXiv  Detail & Related papers  (2024-11-15T18:59:27Z)
• LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch [16.174567164068037]
  We propose a unified learning-based framework called LLMOPT to boost optimization generalization.<n>LLMOPT constructs the introduced five-element formulation as a universal model for learning to define diverse optimization problem types.<n>LLMOPT is able to model various optimization problem types such as linear/nonlinear programming mixed integer programming.
  arXiv  Detail & Related papers  (2024-10-17T04:37:37Z)
• Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization [22.317176475276725]
  We investigate two remarkable phenomena observed during the fine-tuning of Large Language Models (LLMs) Fine-tuning only the $mathbfW_q$ and $mathbfW_v$ matrix significantly improves performance over optimizing the $mathbfW_k$ matrix. We propose a new strategy that improves fine-tuning efficiency in terms of both storage and time.
  arXiv  Detail & Related papers  (2024-10-03T06:37:37Z)
• OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling [62.19438812624467]
  Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning. We propose OptiBench, a benchmark for End-to-end optimization problem-solving with human-readable inputs and outputs.
  arXiv  Detail & Related papers  (2024-07-13T13:27:57Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)
• LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation [26.975412742800614]
  We introduce LLaMoCo, the first instruction-tuning framework designed to adapt large language models for solving optimization problems in a code-to-code manner. Specifically, we establish a comprehensive instruction set containing well-described problem prompts and effective optimization codes. Experiment results demonstrate that a CodeGen (350M) model fine-tuned by our LLaMoCo achieves superior optimization performance compared to GPT-4 Turbo.
  arXiv  Detail & Related papers  (2024-03-02T08:21:59Z)
• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark [166.40879020706151]
  This paper proposes a shift towards BP-free, zeroth-order (ZO) optimization as a solution for reducing memory costs during fine-tuning. Unlike traditional ZO-SGD methods, our work expands the exploration to a wider array of ZO optimization techniques. Our study unveils previously overlooked optimization principles, highlighting the importance of task alignment, the role of the forward gradient method, and the balance between algorithm complexity and fine-tuning performance.
  arXiv  Detail & Related papers  (2024-02-18T14:08:48Z)
• DrugAssist: A Large Language Model for Molecule Optimization [29.95488215594247]
  DrugAssist is an interactive molecule optimization model that performs optimization through human-machine dialogue. DrugAssist has achieved leading results in both single and multiple property optimization. We publicly release a large instruction-based dataset called MolOpt-Instructions for fine-tuning language models on molecule optimization tasks.
  arXiv  Detail & Related papers  (2023-12-28T10:46:56Z)
• Large Language Models as Optimizers [106.52386531624532]
  We propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as prompts. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values. We demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks.
  arXiv  Detail & Related papers  (2023-09-07T00:07:15Z)
• Landscape Surrogate: Learning Decision Losses for Mathematical Optimization Under Partial Information [48.784330281177446]
  Recent works in learning-integrated optimization have shown promise in settings where the optimization is only partially observed or where general-purposes perform poorly without expert tuning. We propose using a smooth and learnable Landscape Surrogate as a replacement for $fcirc mathbfg$. This surrogate, learnable by neural networks, can be computed faster than the $mathbfg$ solver, provides dense and smooth gradients during training, can generalize to unseen optimization problems, and is efficiently learned via alternating optimization.
  arXiv  Detail & Related papers  (2023-07-18T04:29:16Z)
• Enhanced Adaptive Gradient Algorithms for Nonconvex-PL Minimax Optimization [41.28002701420715]
  Minimax optimization has been widely applied in many machine learning tasks. We show that our methods have the best known sample complexity without relying on any specific types.
  arXiv  Detail & Related papers  (2023-03-07T15:33:12Z)
• M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation [145.7321032755538]
  Learning to Optimize (L2O) has drawn increasing attention as it often remarkably accelerates the optimization procedure of complex tasks. This paper investigates a potential solution to this open challenge by meta-training an L2O that can perform fast test-time self-adaptation to an out-of-distribution task.
  arXiv  Detail & Related papers  (2023-02-28T19:23:20Z)
• An Empirical Evaluation of Zeroth-Order Optimization Methods on AI-driven Molecule Optimization [78.36413169647408]
  We study the effectiveness of various ZO optimization methods for optimizing molecular objectives. We show the advantages of ZO sign-based gradient descent (ZO-signGD) We demonstrate the potential effectiveness of ZO optimization methods on widely used benchmark tasks from the Guacamol suite.
  arXiv  Detail & Related papers  (2022-10-27T01:58:10Z)
• Learning to extrapolate using continued fractions: Predicting the critical temperature of superconductor materials [5.905364646955811]
  In the field of Artificial Intelligence (AI) and Machine Learning (ML), the approximation of unknown target functions $y=f(mathbfx)$ is a common objective. We refer to $S$ as the training set and aim to identify a low-complexity mathematical model that can effectively approximate this target function for new instances $mathbfx$.
  arXiv  Detail & Related papers  (2020-11-27T04:57:40Z)
• Stochastic Flows and Geometric Optimization on the Orthogonal Group [52.50121190744979]
  We present a new class of geometrically-driven optimization algorithms on the orthogonal group $O(d)$. We show that our methods can be applied in various fields of machine learning including deep, convolutional and recurrent neural networks, reinforcement learning, flows and metric learning.
  arXiv  Detail & Related papers  (2020-03-30T15:37:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.