Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model

• URL: http://arxiv.org/abs/2401.02051v3
• Date: Sat, 1 Jun 2024 16:48:37 GMT
• Title: Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model
• Authors: Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, Qingfu Zhang,
• Abstract summary: EoH represents the ideas of thoughts in natural language, termed thoughts. They are translated into executable codes by Large Language Models (LLMs) EoH significantly outperforms widely-used human hand-crafted baseline algorithms for the online bin packing problem.
• Score: 22.64392837434924
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Heuristics are widely used for dealing with complex search and optimization problems. However, manual design of heuristics can be often very labour extensive and requires rich working experience and knowledge. This paper proposes Evolution of Heuristic (EoH), a novel evolutionary paradigm that leverages both Large Language Models (LLMs) and Evolutionary Computation (EC) methods for Automatic Heuristic Design (AHD). EoH represents the ideas of heuristics in natural language, termed thoughts. They are then translated into executable codes by LLMs. The evolution of both thoughts and codes in an evolutionary search framework makes it very effective and efficient for generating high-performance heuristics. Experiments on three widely studied combinatorial optimization benchmark problems demonstrate that EoH outperforms commonly used handcrafted heuristics and other recent AHD methods including FunSearch. Particularly, the heuristic produced by EoH with a low computational budget (in terms of the number of queries to LLMs) significantly outperforms widely-used human hand-crafted baseline algorithms for the online bin packing problem.

Related papers

• Beyond the Hype: Benchmarking LLM-Evolved Heuristics for Bin Packing [0.1843404256219181]
  An escalating arms race is both rapidly producing news and improving the efficiency of the processes evolving them. We show that most of the LLMs do not generalise well when evaluated across a broad range of benchmarks. We suggest that any gains from generating very specialist Couplings that only work in small areas of the instance space need to be weighed carefully.
  arXiv  Detail & Related papers  (2025-01-20T11:23:50Z)
• Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Design [33.58608225370497]
  Large Language Model (LLM)-based automatic design (AHD) methods have shown promise in generating high-quality designs without manual intervention. This paper proposes to use Monte Carlo Tree Search (MCTS) for evolutionary evolution.
  arXiv  Detail & Related papers  (2025-01-15T06:00:50Z)
• CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval [103.116634967815]
  We introduce CodeXEmbed, a family of large-scale code embedding models ranging from 400M to 7B parameters. Our novel training pipeline unifies multiple programming languages and transforms various code-related tasks into a common retrieval framework. Our 7B model sets a new state-of-the-art (SOTA) in code retrieval, outperforming the previous leading model, Voyage-Code, by over 20% on CoIR benchmark.
  arXiv  Detail & Related papers  (2024-11-19T16:54:45Z)
• Multi-objective Evolution of Heuristic Using Large Language Model [29.337470185034555]
  We model the search as a multi-objective optimization problem and consider introducing additional practical criteria beyond optimal performance. We propose the first multi-objective search framework, Multi-objective Evolution of Heuristic (MEoH)
  arXiv  Detail & Related papers  (2024-09-25T12:32:41Z)
• Guided Evolution with Binary Discriminators for ML Program Search [64.44893463120584]
  We propose guiding evolution with a binary discriminator, trained online to distinguish which program is better given a pair of programs. We demonstrate our method can speed up evolution across a set of diverse problems including a 3.7x speedup on the symbolic search for MLs and a 4x speedup for RL loss functions.
  arXiv  Detail & Related papers  (2024-02-08T16:59:24Z)
• ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution [35.39046514910755]
  This paper introduces Language Hyper-Heuristics (LHHs), an emerging variant of Hyper-Heuristics, featuring minimal manual intervention and open-ended spaces. To empower LHHs, we presentive Evolution (ReEvo), a novel integration of evolutionary search for efficiently exploring the white space, and reflections to provide verbal gradients within the space.
  arXiv  Detail & Related papers  (2024-02-02T05:04:51Z)
• Algorithm Evolution Using Large Language Model [18.03090066194074]
  We propose a novel approach called Evolution Algorithm using Large Language Model (AEL) AEL does algorithm-level evolution without model training. Human effort and requirements for domain knowledge can be significantly reduced.
  arXiv  Detail & Related papers  (2023-11-26T09:38:44Z)
• Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers [70.18534453485849]
  EvoPrompt is a framework for discrete prompt optimization. It borrows the idea of evolutionary algorithms (EAs) as they exhibit good performance and fast convergence. It significantly outperforms human-engineered prompts and existing methods for automatic prompt generation.
  arXiv  Detail & Related papers  (2023-09-15T16:50:09Z)
• When Do Program-of-Thoughts Work for Reasoning? [51.2699797837818]
  We propose complexity-impacted reasoning score (CIRS) to measure correlation between code and reasoning abilities. Specifically, we use the abstract syntax tree to encode the structural information and calculate logical complexity. Code will be integrated into the EasyInstruct framework at https://github.com/zjunlp/EasyInstruct.
  arXiv  Detail & Related papers  (2023-08-29T17:22:39Z)
• Searching for More Efficient Dynamic Programs [61.79535031840558]
  We describe a set of program transformations, a simple metric for assessing the efficiency of a transformed program, and a search procedure to improve this metric. We show that in practice, automated search can find substantial improvements to the initial program.
  arXiv  Detail & Related papers  (2021-09-14T20:52:55Z)
• Towards Optimally Efficient Tree Search with Deep Learning [76.64632985696237]
  This paper investigates the classical integer least-squares problem which estimates signals integer from linear models. The problem is NP-hard and often arises in diverse applications such as signal processing, bioinformatics, communications and machine learning. We propose a general hyper-accelerated tree search (HATS) algorithm by employing a deep neural network to estimate the optimal estimation for the underlying simplified memory-bounded A* algorithm.
  arXiv  Detail & Related papers  (2021-01-07T08:00:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.