BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks

• URL: http://arxiv.org/abs/2605.17000v1
• Date: Sat, 16 May 2026 13:53:44 GMT
• Title: BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks
• Abstract summary: Black-box optimization methods offer a promising yet underexplored direction for principled, sample-efficient methods.<n>New methods are often only evaluated on synthetic test functions and small-scale datasets.<n>We introduce BoLT, the first LLM-centric benchmark that democratizes research for the BBO community.
• Score: 42.78958516953461
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is often approached heuristically in practice, leading to potentially suboptimal outcomes. By framing them as noisy, expensive, and derivative-free optimization problems, Bayesian optimization (BO) and other black-box optimization (BBO) methods offer a promising yet underexplored direction for principled, sample-efficient methods. However, LLM training and inference costs are prohibitively high for most of the BBO research community, and new methods are often only evaluated on synthetic test functions and small-scale datasets that fail to capture the challenges of modern LLM optimization problems. This impedes the development of BBO methods and makes it difficult to assess their effectiveness on modern LLM tasks. We introduce BoLT, the first LLM-centric benchmark that democratizes LLM research for the BBO community. BoLT is released at https://github.com/chewwt/bolt. BoLT covers broad and well-motivated LLM optimization problems, involving multi-fidelity, multi-objective, heteroscedastic noise, and high-dimensional search spaces. Each problem in BoLT is grounded in real experimental data and made fully reproducible and accessible through lightweight surrogate models fitted to the results of thousands of real LLM experiments. We benchmark BoLT against an extensive range of BO and BBO methods, showing that selected BO methods consistently outperform others across tasks and highlighting gaps in existing BBO methods on LLM tasks, underscoring the need to modernize benchmarks for the BBO community.

Related papers

• LABO: LLM-Accelerated Bayesian Optimization through Broad Exploration and Selective Experimentation [18.30782195527165]
  Large language models (LLMs) are knowledge-driven components in Bayesian optimization.<n>We propose a framework that combines LLM predictions with experimental observations within a single BO loop.<n>We show that LABO consistently outperforms existing methods under identical experimental budgets.
  arXiv  Detail & Related papers  (2026-05-21T06:42:47Z)
• LB-MCTS: Synergizing Large Language Models and Bayesian Optimization for Efficient CASH [10.806107994473495]
  We propose LB-MCTS, a framework synergizing Large Language Models (LLMs) and BO within a Monte Carlo Tree Search structure.<n>It maximizes reasoning with Selective Tuning Memory (STM) and explicit exploration-exploitation trade-off.<n>Experiments on 104 AMLB datasets demonstrate the superiority of LB-MCTS over the competitive baselines.
  arXiv  Detail & Related papers  (2026-01-18T11:26:31Z)
• ChemBOMAS: Accelerated BO in Chemistry with LLM-Enhanced Multi-Agent System [72.63341091857959]
  We introduce ChemBOMAS: a large language model (LLM)-enhanced multi-agent system that accelerates Bayesian optimization.<n>Data-driven strategy involves an 8B-scale LLM regressor fine-tuned on a mere 1% labeled samples.<n>The knowledge-driven strategy employs a hybrid Retrieval-Augmented Generation approach to guide LLM in dividing the search space.<n>ChemBOMAS set a new state-of-the-art, accelerating optimization efficiency by up to 5-fold compared to baseline methods.
  arXiv  Detail & Related papers  (2025-09-10T16:24:08Z)
• LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization [5.844783557050259]
  Large Language Models (LLMs) have shown remarkable adaptability in low-data regimes.<n>We propose LLINBO: LLM-in-the-Loop BO, a hybrid framework for BO that combines LLMs with statistical surrogate experts.<n>We end the paper with a real-life proof-of-concept in the context of 3D printing.
  arXiv  Detail & Related papers  (2025-05-20T15:54:48Z)
• Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation [55.21013307734612]
  AoPS-Instruct is a dataset of more than 600,000 high-quality QA pairs.<n>LiveAoPSBench is an evolving evaluation set with timestamps, derived from the latest forum data.<n>Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning.
  arXiv  Detail & Related papers  (2025-01-24T06:39:38Z)
• Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System [75.25394449773052]
  Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving.<n>Yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods.<n>We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness.
  arXiv  Detail & Related papers  (2024-10-10T17:00:06Z)
• A Multi-LLM Debiasing Framework [85.17156744155915]
  Large Language Models (LLMs) are powerful tools with the potential to benefit society immensely, yet, they have demonstrated biases that perpetuate societal inequalities. Recent research has shown a growing interest in multi-LLM approaches, which have been demonstrated to be effective in improving the quality of reasoning. We propose a novel multi-LLM debiasing framework aimed at reducing bias in LLMs.
  arXiv  Detail & Related papers  (2024-09-20T20:24:50Z)
• Large Language Models to Enhance Bayesian Optimization [57.474613739645605]
  We present LLAMBO, a novel approach that integrates the capabilities of Large Language Models (LLM) within Bayesian optimization. At a high level, we frame the BO problem in natural language, enabling LLMs to iteratively propose and evaluate promising solutions conditioned on historical evaluations. Our findings illustrate that LLAMBO is effective at zero-shot warmstarting, and enhances surrogate modeling and candidate sampling, especially in the early stages of search when observations are sparse.
  arXiv  Detail & Related papers  (2024-02-06T11:44:06Z)
• Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward [29.81212051279456]
  Recent advancements in model compression and system-level optimization methods aim to enhance LLM inference. This survey offers an overview of these methods, emphasizing recent developments.
  arXiv  Detail & Related papers  (2024-02-02T06:29:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.