An Open-Source Training Dataset for Foundation Models for Black-box Optimization

• URL: http://arxiv.org/abs/2605.23417v1
• Date: Fri, 22 May 2026 09:27:59 GMT
• Title: An Open-Source Training Dataset for Foundation Models for Black-box Optimization
• Abstract summary: We introduce BBO-Pile, the first open-source dataset comprising over 500K optimization trajectories across 30 different black-boxes.<n>We train a family of foundation models at multiple scales, ranging from 2M to 80M parameters.<n>Our results demonstrate that large-scale pre-training is a viable and effective approach to imitate black-box optimization methods.
• Score: 6.259657082419056
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most black-box optimization methods require extensive hyperparameter tuning, often limiting their ability to generalize across different optimization domains. Foundation models for black-box optimization that learn optimization principles from a large collection of optimization trajectories offer a promising alternative, with the potential to outperform manually designed methods across diverse problem classes. However, prior work has either relied on non-public datasets or on purely synthetic data, limiting reproducibility and generalization to real-world problems. As a result, progress in this area has been constrained by the lack of large-scale, real-world, publicly available pre-training data. We introduce BBO-Pile, the first open-source dataset comprising over 500K optimization trajectories evaluated across 3095 different black-boxes for different optimizers, which represents by far the largest public dataset for this task. Using this dataset, we train a family of foundation models at multiple scales, ranging from 2M to 80M parameters and from 200M to 2B training tokens, and study their scaling behavior with respect to compute. Our results demonstrate that large-scale pre-training is a viable and effective approach to imitate black-box optimization methods, paving the way for future research in this direction.

Related papers

• The Chicken and Egg Dilemma: Co-optimizing Data and Model Configurations for LLMs [86.27977008139435]
  JoBS is an approach that uses a scaling-law-inspired performance predictor to aid Bayesian optimization.<n>We study JoBS's average regret and devise the optimal budget allocation to minimize regret.
  arXiv  Detail & Related papers  (2026-02-09T07:33:40Z)
• ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimization [31.894110383242566]
  We present ZeroShot, a general-purpose, pretrained model for continuous black-box optimization tasks ranging from 2D to 20D.<n>Our approach leverages offline reinforcement learning on large-scale optimization tasks collected from 12 BO variants.
  arXiv  Detail & Related papers  (2025-10-03T14:33:23Z)
• ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization [11.433087692377779]
  We present results for pre-training and instruction finetuning across models ranging from 1 million to 7 billion parameters.<n>We demonstrate consistently strong results relative to a wide range of baselines, resulting inspeed-ups of over 500%.<n>In addition, we broaden access to research by sharing ADMIRE IFT Runs, a dataset of 460 full training & evaluation runs worth over 13,000 GPU hours.
  arXiv  Detail & Related papers  (2025-08-15T15:53:09Z)
• Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection [65.96556073745197]
  DiverSified File selection algorithm (DiSF) is proposed to select the most decorrelated text files in the feature space.<n>DiSF saves 98.5% of 590M training files in SlimPajama, outperforming the full-data pre-training within a 50B training budget.
  arXiv  Detail & Related papers  (2025-04-29T11:13:18Z)
• Posterior Inference with Diffusion Models for High-dimensional Black-box Optimization [17.92257026306603]
  generative models have emerged to solve black-box optimization problems.<n>We introduce textbfDiBO, a novel framework for solving high-dimensional black-box optimization problems.<n>Our method outperforms state-of-the-art baselines across synthetic and real-world tasks.
  arXiv  Detail & Related papers  (2025-02-24T04:19:15Z)
• Functional Graphical Models: Structure Enables Offline Data-Driven Optimization [111.28605744661638]
  We show how structure can enable sample-efficient data-driven optimization. We also present a data-driven optimization algorithm that infers the FGM structure itself.
  arXiv  Detail & Related papers  (2024-01-08T22:33:14Z)
• VeLO: Training Versatile Learned Optimizers by Scaling Up [67.90237498659397]
  We leverage the same scaling approach behind the success of deep learning to learn versatiles. We train an ingest for deep learning which is itself a small neural network that ingests and outputs parameter updates. We open source our learned, meta-training code, the associated train test data, and an extensive benchmark suite with baselines at velo-code.io.
  arXiv  Detail & Related papers  (2022-11-17T18:39:07Z)
• Pre-training helps Bayesian optimization too [49.28382118032923]
  We seek an alternative practice for setting functional priors. In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori. Our results show that our method is able to locate good hyper parameters at least 3 times more efficiently than the best competing methods.
  arXiv  Detail & Related papers  (2022-07-07T04:42:54Z)
• Conservative Objective Models for Effective Offline Model-Based Optimization [78.19085445065845]
  Computational design problems arise in a number of settings, from synthetic biology to computer architectures. We propose a method that learns a model of the objective function that lower bounds the actual value of the ground-truth objective on out-of-distribution inputs. COMs are simple to implement and outperform a number of existing methods on a wide range of MBO problems.
  arXiv  Detail & Related papers  (2021-07-14T17:55:28Z)
• Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020 [10.782941183498851]
  This paper presents the results and insights from the black-box optimization challenge at NeurIPS 2020. This was the first black-box optimization challenge with a machine learning emphasis.
  arXiv  Detail & Related papers  (2021-04-20T18:44:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.