Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch

• URL: http://arxiv.org/abs/2603.24647v2
• Date: Sun, 29 Mar 2026 18:46:53 GMT
• Title: Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch
• Abstract summary: We show that an LLM agent that edits training source code in an unconstrained search space narrows the gap to classical methods.<n>We introduce Centaur, a hybrid that shares CMA-ES's internal state, including mean vector, step-size, and covariance matrix.<n> Centaur achieves the best result in our experiments, with its 0.8B variant outperforming the 27B variant.
• Score: 42.242102214102566
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The autoresearch repository enables an LLM agent to search for optimal hyperparameter configurations on an unconstrained search space by editing the training code directly. Given a fixed compute budget and constraints, we use autoresearch as a testbed to compare classical hyperparameter optimization (HPO) algorithms against LLM-based methods on tuning the hyperparameters of a small language model. Within a fixed hyperparameter search space, classical HPO methods such as CMA-ES and TPE consistently outperform LLM-based agents. However, an LLM agent that directly edits training source code in an unconstrained search space narrows the gap to classical methods substantially despite using only a self-hosted open-weight 27B model. Methods that avoid out-of-memory failures outperform those with higher search diversity, suggesting that reliability matters more than exploration breadth. While small and mid-sized LLMs struggle to track optimization state across trials, classical methods lack domain knowledge. To bridge this gap, we introduce Centaur, a hybrid that shares CMA-ES's internal state, including mean vector, step-size, and covariance matrix, with an LLM. Centaur achieves the best result in our experiments, with its 0.8B variant outperforming the 27B variant, suggesting that a cheap LLM suffices when paired with a strong classical optimizer. The 0.8B model is insufficient for unconstrained code editing but sufficient for hybrid optimization, while scaling to 27B provides no advantage for fixed search space methods. Preliminary experiments with the frontier model Gemini 3.1 Pro Preview do not close the gap to classical methods. Code is available at https://github.com/ferreirafabio/autoresearch-automl.

Related papers

• A Hybrid Nested Harness for Decoupling Structure and Parameters in LLM-Driven Optimization [6.599344783327054]
  In evolutionary algorithms powered by language models, the LLM acts as a single operator that simultaneously updates structural components and continuous parameters.<n>We formalize a hybrid nested search, in which an outer loop has the LLM propose a structural sketch, with numeric gaps, and an innernest tunes the sketch.<n>We validate our framework across three scientific domains: (i) meta-optimizers on closed-form test functions, (ii) code-based policies for systems research and social dilemmas, and (iii) approximate Bayesian inference tasks.
  arXiv  Detail & Related papers  (2026-08-08T14:35:22Z)
• Better Together, in the Right Order: Classical-then-LLM Optimization for SE [4.168157981135697]
  A growing body of work combines large language models (LLMs) with classicals for software engineering (SE) configuration tasks.<n>We report that there is much value in seeding an LLM with the results from a cheap classical learner.<n> Applied to over 100 SNAP SE tasks, it is the single best of all methods studied, reaching the top tier on 85% of tasks ahead of the same LLM run alone (75%) and ahead of every method in which the classical retains control.
  arXiv  Detail & Related papers  (2026-07-01T02:32:14Z)
• GRIMIP: A General Framework for Instance-Specific Configuration of MIP Solvers Using LLMs [9.2939109839777]
  This paper introduces textbfGRIMIP (textbfunderlineGeneral textbfunderlineReasoning for textbfunderlineInstance-specific textbfunderlineMIP configuration), a novel hybrid intelligence framework.
  arXiv  Detail & Related papers  (2026-06-22T13:10:50Z)
• $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• How to Train Your LLM Web Agent: A Statistical Diagnosis [96.86317871461834]
  We present the first statistically grounded study on compute allocation for LLM web-agent post-training.<n>Our approach uses a two-stage pipeline, training a Llama 3.1 8B student to imitate a Llama 3.3 70B teacher via supervised fine-tuning (SFT) and on-policy reinforcement learning.<n>Our results show that combining SFT with on-policy RL consistently outperforms either approach alone on both WorkArena and MiniWob++.
  arXiv  Detail & Related papers  (2025-07-05T17:12:33Z)
• LLM Agent for Hyper-Parameter Optimization [27.801667344330944]
  In this paper, we design an Large Language Model (LLM) agent for automatic hyper-parameters-tuning.<n>Our experiment results show that the minimal sum-rate achieved by hyper-parameters generated via our LLM agent is significantly higher than those by both humans and random generation methods.
  arXiv  Detail & Related papers  (2025-06-18T06:28:22Z)
• LLM-First Search: Self-Guided Exploration of the Solution Space [29.780554400938335]
  Large Language Models (LLMs) have demonstrated remarkable improvements in reasoning and planning through increased test-time compute.<n>We propose textbfLLM-First Search (LFS), a novel textitLLM Self-Guided Search method.
  arXiv  Detail & Related papers  (2025-06-05T16:27:49Z)
• Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning? [45.58422897857411]
  This work explores the use of large language models (LLMs) for hyperparameter optimization by fine-tuning a parameter-efficient version of Code Llama using LoRA.<n>Our approach achieves competitive or superior Root Mean Square Error (RMSE) while substantially reducing computational overhead.<n>Results demonstrate that LLM-based optimization not only rivals established Bayesian methods like Tree-structured Parzen Estimators (TPE) but also accelerates tuning for real-world applications requiring perceptual quality and low-latency processing.
  arXiv  Detail & Related papers  (2025-04-08T13:15:47Z)
• Training Deep Learning Models with Norm-Constrained LMOs [56.00317694850397]
  We propose a new family of algorithms that uses the linear minimization oracle (LMO) to adapt to the geometry of the problem.<n>We demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam.
  arXiv  Detail & Related papers  (2025-02-11T13:10:34Z)
• Search for Efficient Large Language Models [52.98684997131108]
  Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Weight pruning, quantization, and distillation have been embraced to compress LLMs, targeting memory reduction and inference acceleration. Most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures.
  arXiv  Detail & Related papers  (2024-09-25T21:32:12Z)
• Scaling Sparse Fine-Tuning to Large Language Models [67.59697720719672]
  Large Language Models (LLMs) are difficult to fully fine-tune due to their sheer number of parameters. We propose SpIEL, a novel sparse finetuning method which maintains an array of parameter indices and the deltas of these parameters relative to their pretrained values. We show that SpIEL is superior to popular parameter-efficient fine-tuning methods like LoRA in terms of performance and comparable in terms of run time.
  arXiv  Detail & Related papers  (2024-01-29T18:43:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.