Improving Existing Optimization Algorithms with LLMs

• URL: http://arxiv.org/abs/2502.08298v1
• Date: Wed, 12 Feb 2025 10:58:57 GMT
• Title: Improving Existing Optimization Algorithms with LLMs
• Authors: Camilo Chacón Sartori, Christian Blum,
• Abstract summary: This paper investigates how Large Language Models (LLMs) can enhance existing optimization algorithms. Using their pre-trained knowledge, we demonstrate their ability to propose innovative variations and implementation strategies. Our results show that an alternative proposed by GPT-4o outperforms the expert-designed of CMSA.
• Score: 0.9668407688201361
• License:
• Abstract: The integration of Large Language Models (LLMs) into optimization has created a powerful synergy, opening exciting research opportunities. This paper investigates how LLMs can enhance existing optimization algorithms. Using their pre-trained knowledge, we demonstrate their ability to propose innovative heuristic variations and implementation strategies. To evaluate this, we applied a non-trivial optimization algorithm, Construct, Merge, Solve and Adapt (CMSA) -- a hybrid metaheuristic for combinatorial optimization problems that incorporates a heuristic in the solution construction phase. Our results show that an alternative heuristic proposed by GPT-4o outperforms the expert-designed heuristic of CMSA, with the performance gap widening on larger and denser graphs. Project URL: https://imp-opt-algo-llms.surge.sh/

Related papers

• Large Language Models for Combinatorial Optimization of Design Structure Matrix [4.513609458468522]
  Combinatorial optimization (CO) is essential for improving efficiency and performance in engineering applications. When it comes to real-world engineering problems, algorithms based on pure mathematical reasoning are limited and incapable to capture the contextual nuances necessary for optimization. This study explores the potential of Large Language Models (LLMs) in solving engineering CO problems by leveraging their reasoning power and contextual knowledge.
  arXiv  Detail & Related papers  (2024-11-19T15:39:51Z)
• Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithms [3.833708891059351]
  Large Language Models (LLMs) and Evolutionary Algorithms (EAs) offer promising new approach to overcome limitations and make optimization more automated. LLMs act as dynamic agents that can generate, refine, and interpret optimization strategies. EAs efficiently explore complex solution spaces through evolutionary operators.
  arXiv  Detail & Related papers  (2024-10-28T09:04:49Z)
• A Problem-Oriented Perspective and Anchor Verification for Code Optimization [43.28045750932116]
  Large language models (LLMs) have shown remarkable capabilities in solving various programming tasks. This paper investigates the capabilities of LLMs in optimizing code for minimal execution time.
  arXiv  Detail & Related papers  (2024-06-17T16:10:10Z)
• Discovering Preference Optimization Algorithms with and for Large Language Models [50.843710797024805]
  offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. We perform objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention. Experiments demonstrate the state-of-the-art performance of DiscoPOP, a novel algorithm that adaptively blends logistic and exponential losses.
  arXiv  Detail & Related papers  (2024-06-12T16:58:41Z)
• LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning [69.95292905263393]
  We show that gradient-based and high-level LLMs can effectively collaborate a combined optimization framework. In this paper, we show that these complementary to each other and can effectively collaborate a combined optimization framework.
  arXiv  Detail & Related papers  (2024-05-30T06:24:14Z)
• When Large Language Model Meets Optimization [7.822833805991351]
  Large language models (LLMs) facilitate intelligent modeling and strategic decision-making in optimization. This review outlines the progress and potential of combining LLMs with optimization algorithms.
  arXiv  Detail & Related papers  (2024-05-16T13:54:37Z)
• An Empirical Evaluation of Zeroth-Order Optimization Methods on AI-driven Molecule Optimization [78.36413169647408]
  We study the effectiveness of various ZO optimization methods for optimizing molecular objectives. We show the advantages of ZO sign-based gradient descent (ZO-signGD) We demonstrate the potential effectiveness of ZO optimization methods on widely used benchmark tasks from the Guacamol suite.
  arXiv  Detail & Related papers  (2022-10-27T01:58:10Z)
• Optimizer Amalgamation [124.33523126363728]
  We are motivated to study a new problem named Amalgamation: how can we best combine a pool of "teacher" amalgamations into a single "student" that can have stronger problem-specific performance? First, we define three differentiable mechanisms to amalgamate a pool of analyticals by gradient descent. In order to reduce variance of the process, we also explore methods to stabilize the process by perturbing the target.
  arXiv  Detail & Related papers  (2022-03-12T16:07:57Z)
• Optimization-Inspired Learning with Architecture Augmentations and Control Mechanisms for Low-Level Vision [74.9260745577362]
  This paper proposes a unified optimization-inspired learning framework to aggregate Generative, Discriminative, and Corrective (GDC) principles. We construct three propagative modules to effectively solve the optimization models with flexible combinations. Experiments across varied low-level vision tasks validate the efficacy and adaptability of GDC.
  arXiv  Detail & Related papers  (2020-12-10T03:24:53Z)
• Bilevel Optimization: Convergence Analysis and Enhanced Design [63.64636047748605]
  Bilevel optimization is a tool for many machine learning problems. We propose a novel stoc-efficientgradient estimator named stoc-BiO.
  arXiv  Detail & Related papers  (2020-10-15T18:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.