An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models

• URL: http://arxiv.org/abs/2601.12723v2
• Date: Fri, 23 Jan 2026 04:30:45 GMT
• Title: An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models
• Authors: Yuhiro Ono, Tomohiro Harada, Yukiya Miura,
• Abstract summary: We propose an evolutionary automatic benchmark generation framework that leverages a large language model (LLM) as a generative operator.<n>In this framework, the LLM serves as an evolutionary operator that generates and evolves benchmark problems within a flexible, expressive representation space.<n> Experimental results show that LLM-EBG successfully produces benchmark problems in which the designated target algorithm consistently outperforms the comparative algorithm in more than 80% of trials.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Optimization benchmarks play a fundamental role in assessing algorithm performance; however, existing artificial benchmarks often fail to capture the diversity and irregularity of real-world problem structures, while benchmarks derived from real-world problems are costly and difficult to construct. To address these challenges, we propose an evolutionary automatic benchmark generation framework that leverages a large language model (LLM) as a generative operator, termed the LLM-driven evolutionary benchmark generator (LLM-EBG). In this framework, the LLM serves as an evolutionary operator that generates and evolves benchmark problems within a flexible, expressive representation space. As a case study, we generate unconstrained single-objective continuous minimization problems represented as mathematical expressions designed to induce significant performance differences between a genetic algorithm (GA) and differential evolution (DE). Experimental results show that LLM-EBG successfully produces benchmark problems in which the designated target algorithm consistently outperforms the comparative algorithm in more than 80\% of trials. Furthermore, exploratory landscape analysis reveals that benchmarks favoring GA are highly sensitive to variable scaling, demonstrating that the proposed framework can generate problems with distinct geometric characteristics that reflect the intrinsic search behaviors of different optimization algorithms.

Related papers

• TIDE: Tuning-Integrated Dynamic Evolution for LLM-Based Automated Heuristic Design [7.264986493460248]
  TIDE is a Tuning-Integrated Dynamic Evolution framework designed to decouple structural reasoning from parameter optimization.<n> experiments across nine optimization problems demonstrate that TIDE significantly outperforms state-of-the-art tuning methods.
  arXiv  Detail & Related papers  (2026-01-29T04:00:02Z)
• GLOW: Graph-Language Co-Reasoning for Agentic Workflow Performance Prediction [51.83437071408662]
  We propose GLOW, a unified framework for AW performance prediction.<n>GLOW combines the graph-structure modeling capabilities of GNNs with the reasoning power of LLMs.<n>Experiments on FLORA-Bench show that GLOW outperforms state-of-the-art baselines in prediction accuracy and ranking utility.
  arXiv  Detail & Related papers  (2025-12-11T13:30:46Z)
• ORGEval: Graph-Theoretic Evaluation of LLMs in Optimization Modeling [18.8099769877788]
  ORGEval is a graph-theoretic evaluation framework for assessing Large Language Models' capabilities in formulating linear and mixed-integer linear programs.<n>We show that ORGEval can successfully detect model equivalence and produce 100% consistent results across random parameter configurations.<n>Our results reveal that although optimization modeling remains challenging for all LLMs, DeepSeek-V3 and Claude-Opus-4 achieve the highest accuracies under direct prompting.
  arXiv  Detail & Related papers  (2025-10-31T16:35:52Z)
• Graph Neural Network Assisted Genetic Algorithm for Structural Dynamic Response and Parameter Optimization [1.5383027029023142]
  optimization of structural parameters, such as mass(m), stiffness(k), and damping coefficient(c) is critical for designing efficient, resilient, and stable structures.<n>This study proposes a hybrid data-driven framework that integrates a Graph Neural Network (GNN) surrogate model with a Genetic Algorithm (GA) to overcome these challenges.
  arXiv  Detail & Related papers  (2025-10-26T21:14:59Z)
• Dynamic Generation of Multi-LLM Agents Communication Topologies with Graph Diffusion Models [99.85131798240808]
  We introduce a novel generative framework called textitGuided Topology Diffusion (GTD)<n>Inspired by conditional discrete graph diffusion models, GTD formulates topology synthesis as an iterative construction process.<n>At each step, the generation is steered by a lightweight proxy model that predicts multi-objective rewards.<n>Experiments show that GTD can generate highly task-adaptive, sparse, and efficient communication topologies.
  arXiv  Detail & Related papers  (2025-10-09T05:28:28Z)
• Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery [2.0958996366233094]
  We investigate meta-heuristic optimisation algorithms automatically generated by Large Language Model driven algorithm discovery methods.<n>We iteratively evolve black-box optimisations, evaluated on 10 functions from the BBOB benchmark suite.<n>We log behavioural metrics including exploration, exploitation, convergence and stagnation measures, for each run, and analyse these via visual projections and network-based representations.
  arXiv  Detail & Related papers  (2025-07-04T14:19:39Z)
• EALG: Evolutionary Adversarial Generation of Language Model-Guided Generators for Combinatorial Optimization [5.575239967310329]
  We introduce EALG (Evolutionary Adrial Generation of Language Model Generators), a novel framework that co-evolutione optimization problem instances and their corresponding solvers using large language models (LLMs)<n>EALG leverages a mutation-based approach that dynamically evolves instance generation procedures to create increasingly difficult problems, while simultaneously adaptive adversarial algorithms through interactions with LLMs guided by algorithmic structure.<n>This work explores a new paradigm for optimization that integrates instance generation with solver design, resulting in state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-06-03T08:13:41Z)
• Large Language Model-Based Benchmarking Experiment Settings for Evolutionary Multi-Objective Optimization [5.072077366588175]
  We try to examine the implicit assumption about the performance comparison of EMO algorithms in large language models.<n>Our experiments show that LLMs often suggest classical benchmark settings.
  arXiv  Detail & Related papers  (2025-02-28T14:46:34Z)
• Revisiting GANs by Best-Response Constraint: Perspective, Methodology, and Application [49.66088514485446]
  Best-Response Constraint (BRC) is a general learning framework to explicitly formulate the potential dependency of the generator on the discriminator. We show that even with different motivations and formulations, a variety of existing GANs ALL can be uniformly improved by our flexible BRC methodology.
  arXiv  Detail & Related papers  (2022-05-20T12:42:41Z)
• Evolving Pareto-Optimal Actor-Critic Algorithms for Generalizability and Stability [67.8426046908398]
  Generalizability and stability are two key objectives for operating reinforcement learning (RL) agents in the real world. This paper presents MetaPG, an evolutionary method for automated design of actor-critic loss functions.
  arXiv  Detail & Related papers  (2022-04-08T20:46:16Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)
• EOS: a Parallel, Self-Adaptive, Multi-Population Evolutionary Algorithm for Constrained Global Optimization [68.8204255655161]
  EOS is a global optimization algorithm for constrained and unconstrained problems of real-valued variables. It implements a number of improvements to the well-known Differential Evolution (DE) algorithm. Results prove that EOSis capable of achieving increased performance compared to state-of-the-art single-population self-adaptive DE algorithms.
  arXiv  Detail & Related papers  (2020-07-09T10:19:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.