Better Together, in the Right Order: Classical-then-LLM Optimization for SE

• URL: http://arxiv.org/abs/2607.02583v1
• Date: Wed, 01 Jul 2026 02:32:14 GMT
• Title: Better Together, in the Right Order: Classical-then-LLM Optimization for SE
• Abstract summary: A growing body of work combines large language models (LLMs) with classicals for software engineering (SE) configuration tasks.<n>We report that there is much value in seeding an LLM with the results from a cheap classical learner.<n> Applied to over 100 SNAP SE tasks, it is the single best of all methods studied, reaching the top tier on 85% of tasks ahead of the same LLM run alone (75%) and ahead of every method in which the classical retains control.
• Score: 4.168157981135697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A growing body of work combines large language models (LLMs) with classical optimizers for software engineering (SE) configuration tasks. Often, the classical optimizer is in charge: it owns the search loop and calls the LLM only to assist in subroutines (e.g. to warm-start the first generation, propose a mutation, or stand in as a surrogate). We report that there is much value in the reverse approach: seeding an LLM with the results from a cheap classical learner. We call this method SNAP2. Applied to over 100 SE tasks, it is the single best of all methods studied, reaching the top tier on 85% of tasks, ahead of the same LLM run alone (75%) and ahead of every method in which the classical optimizer retains control. It is also less expensive: relative to the LLM-alone method, it uses roughly 30% fewer tokens and runs 1.4x faster, since the classical setup performs the inexpensive work, and the LLM is invoked only to finish. We conclude that it is unwise to study classical learners or LLMs in isolation: there is much value in combining the two, and in the order that combination is applied.

Related papers

• Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch [42.242102214102566]
  We show that an LLM agent that edits training source code in an unconstrained search space narrows the gap to classical methods.<n>We introduce Centaur, a hybrid that shares CMA-ES's internal state, including mean vector, step-size, and covariance matrix.<n> Centaur achieves the best result in our experiments, with its 0.8B variant outperforming the 27B variant.
  arXiv  Detail & Related papers  (2026-03-25T17:29:40Z)
• Slm-mux: Orchestrating small language models for reasoning [52.461958665375896]
  We propose a three-stage approach for orchestrating small language models (SLMs)<n>First, we introduce SLM-MUX, a multi-model architecture that effectively coordinates multiple SLMs.<n>With just two SLMS, SLM-MUX outperforms Qwen 2.5 72B on GPQA and GSM8K, and matches its performance on MATH.
  arXiv  Detail & Related papers  (2025-10-06T17:49:58Z)
• Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs [22.39397810186991]
  We propose ZO Fine-tuner, a learning-based zeroth-order for large language models.<n>It automatically learns efficient perturbation strategies through a compact and memory-efficient design.<n>Experiments show that ZO Fine-tuner outperforms prior zeroth-order baselines in 82.1% of task-model combinations.
  arXiv  Detail & Related papers  (2025-10-01T02:01:07Z)
• Open-Source LLMs Collaboration Beats Closed-Source LLMs: A Scalable Multi-Agent System [51.04535721779685]
  This paper aims to demonstrate the potential and strengths of open-source collectives.<n>We propose SMACS, a scalable multi-agent collaboration system (MACS) framework with high performance.<n> Experiments on eight mainstream benchmarks validate the effectiveness of our SMACS.
  arXiv  Detail & Related papers  (2025-07-14T16:17:11Z)
• When Transformers Meet Recommenders: Integrating Self-Attentive Sequential Recommendation with Fine-Tuned LLMs [0.0]
  SASRecLLM is a novel framework that integrates SASRec as a collaborative encoder with an LLM fine-tuned using Low-Rank Adaptation (LoRA)<n>Experiments on multiple datasets demonstrate that SASRecLLM achieves robust and consistent improvements over strong baselines in both cold-start and warm-start scenarios.
  arXiv  Detail & Related papers  (2025-07-08T07:26:55Z)
• STAR: A Simple Training-free Approach for Recommendations using Large Language Models [36.18841135511487]
  Current state-of-the-art methods rely on fine-tuning large language models (LLMs) to achieve optimal results.<n>We propose a framework that utilizes LLMs and can be applied to various recommendation tasks without the need for fine-tuning.<n>Our method achieves Hits@10 performance of +23.8% on Beauty, +37.5% on Toys & Games, and -1.8% on Sports & Outdoors.
  arXiv  Detail & Related papers  (2024-10-21T19:34:40Z)
• Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning - A Convex Optimization Perspective [55.66517396157806]
  The widely adopted approach in post-training popular open-source LLMs is to sequentially perform SFT and RLHF/DPO.<n>This is suboptimal in terms of SFT and RLHF/DPO trade-off.<n>We propose a practical joint post-training framework which has theoretical convergence guarantees and empirically outperforms sequential post-training framework.
  arXiv  Detail & Related papers  (2024-10-20T19:38:41Z)
• AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models [94.82766517752418]
  We propose AlphaPruning, which uses shape metrics to allocate layerwise sparsity ratios in a more theoretically principled manner. Our results show that AlphaPruning prunes LLaMA-7B to 80% sparsity while maintaining reasonable perplexity, marking a first in the literature on LLMs.
  arXiv  Detail & Related papers  (2024-10-14T03:35:11Z)
• LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity [7.945893812374361]
  We introduce the focal diversity metric to capture the diversity-performance correlation among component LLMs of an ensemble. We develop a diversity-optimized ensemble pruning algorithm to select the top-k sub-ensembles from a pool of $N$ base LLMs. Our pruning method recommends top-performing LLM subensembles of size $S$, often much smaller than $N$.
  arXiv  Detail & Related papers  (2024-10-04T22:31:15Z)
• LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning [69.95292905263393]
  We show that gradient-based and high-level LLMs can effectively collaborate a combined optimization framework.<n>In this paper, we show that these complementary to each other and can effectively collaborate a combined optimization framework.
  arXiv  Detail & Related papers  (2024-05-30T06:24:14Z)
• Value Augmented Sampling for Language Model Alignment and Personalization [39.070662999014836]
  We present a new framework for reward optimization, Value Augmented Sampling (VAS) VAS solves for the optimal reward-maximizing policy without co-training the policy and the value function. Our algorithm unlocks the new capability of composing several rewards and controlling the extent of each one during deployment time.
  arXiv  Detail & Related papers  (2024-05-10T17:59:04Z)
• Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection [80.63946798650653]
  Decision centers on whether to use a large LLM with better performance or a smaller one with reduced costs. We propose a simpler solution; we use only the uncertainty of the generations of the small LLM as the decision criterion. Our experiments reveal this simple solution optimally balances cost and performance, outperforming existing methods on 25 out of 27 experimental setups.
  arXiv  Detail & Related papers  (2024-05-03T14:38:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.