CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization

• URL: http://arxiv.org/abs/2602.09851v1
• Date: Tue, 10 Feb 2026 14:54:17 GMT
• Title: CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization
• Abstract summary: Feature Engineering is pivotal in automated machine learning (AutoML)<n>Existing methods fail to construct free-form FE pipelines, remaining confined to isolated subtasks such as feature generation.<n>We present CoFEH, a collaborative framework that interleaves LLM-based FE and Bayesian HPO for robust end-to-end AutoML.
• Score: 13.416415080328195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Feature Engineering (FE) is pivotal in automated machine learning (AutoML) but remains a bottleneck for traditional methods, which treat it as a black-box search, operating within rigid, predefined search spaces and lacking domain awareness. While Large Language Models (LLMs) offer a promising alternative by leveraging semantic reasoning to generate unbounded operators, existing methods fail to construct free-form FE pipelines, remaining confined to isolated subtasks such as feature generation. Most importantly, they are rarely optimized jointly with hyperparameter optimization (HPO) of the ML model, leading to greedy "FE-then-HPO" workflows that cannot capture strong FE-HPO interactions. In this paper, we present CoFEH, a collaborative framework that interleaves LLM-based FE and Bayesian HPO for robust end-to-end AutoML. CoFEH uses an LLM-driven FE optimizer powered by Tree of Thought (ToT) to explore flexible FE pipelines, a Bayesian optimization (BO) module to solve HPO, and a dynamic optimizer selector that realizes interleaved optimization by adaptively scheduling FE and HPO steps. Crucially, we introduce a mutual conditioning mechanism that shares context between LLM and BO, enabling mutually informed decisions. Experiments show that CoFEH not only outperforms traditional and LLM-based FE baselines, but also achieves superior end-to-end performance under joint optimization.

Related papers

• Execution-Verified Reinforcement Learning for Optimization Modeling [49.171122807323634]
  Execution-verified learning framework treats a mathematical programming solver as a deterministic, interactive verifier.<n>Experiments on NL4OPT, MAMO, IndustryOR, and OptiBench across Gurobi, OR-Tools, and COPT show that EVOM matches or outperforms process-supervised SFT.
  arXiv  Detail & Related papers  (2026-04-01T03:39:11Z)
• A Step Toward Federated Pretraining of Multimodal Large Language Models [87.64508862413565]
  Federated Learning offers a promising solution to unlock distributed resources.<n>Fed-MA is a lightweight pre-training paradigm that freezes the vision encoder and LLM while collaboratively training the cross-modal projector.<n>We propose Fed-CMP, a pioneering framework for federated MLLM pre-training.
  arXiv  Detail & Related papers  (2026-03-25T08:16:23Z)
• ProOPF: Benchmarking and Improving LLMs for Professional-Grade Power Systems Optimization Modeling [26.431357381919295]
  We introduce textbfProOPF-D and textbfProOPF-B, a dataset and benchmark for professional-grade OPF modeling.<n>ProOPF-D contains 12K instances pairing NL requests with parameter adjustments and structural extensions to a canonical OPF, together with executable implementations.<n>ProOPF-B provides 121 expert-annotated test cases with ground-truth code, enabling end-to-end evaluation under both concrete and abstract OPF modeling regimes.
  arXiv  Detail & Related papers  (2026-02-03T03:59:03Z)
• A Meta-Knowledge-Augmented LLM Framework for Hyperparameter Optimization in Time-Series Forecasting [0.0]
  We introduce LLM-AutoOpt, a hybrid HPO framework that combines BO with LLM-based contextual reasoning.<n>We show that LLM-AutoOpt achieves improved predictive performance and more interpretable optimization behavior compared to BO and LLM baselines without meta-knowledge.
  arXiv  Detail & Related papers  (2026-02-01T21:26:57Z)
• Task-free Adaptive Meta Black-box Optimization [55.461814601130044]
  We propose the Adaptive meta Black-box Optimization Model (ABOM), which performs online parameter adaptation using solely optimization data from the target task.<n>Unlike conventional metaBBO frameworks that decouple meta-training and optimization phases, ABOM introduces a closed-loop parameter learning mechanism, where parameterized evolutionary operators continuously self-update.<n>This paradigm shift enables zero-shot optimization: ABOM competitive performance on synthetic BBO benchmarks and realistic unmanned aerial vehicle path planning problems without any handcrafted training tasks.
  arXiv  Detail & Related papers  (2026-01-29T09:54:10Z)
• LB-MCTS: Synergizing Large Language Models and Bayesian Optimization for Efficient CASH [10.806107994473495]
  We propose LB-MCTS, a framework synergizing Large Language Models (LLMs) and BO within a Monte Carlo Tree Search structure.<n>It maximizes reasoning with Selective Tuning Memory (STM) and explicit exploration-exploitation trade-off.<n>Experiments on 104 AMLB datasets demonstrate the superiority of LB-MCTS over the competitive baselines.
  arXiv  Detail & Related papers  (2026-01-18T11:26:31Z)
• LLM4CMO: Large Language Model-aided Algorithm Design for Constrained Multiobjective Optimization [54.35609820607923]
  Large language models (LLMs) offer new opportunities for assisting with algorithm design.<n>We propose LLM4CMO, a novel CMOEA based on a dual-population, two-stage framework.<n>LLMs can serve as efficient co-designers in the development of complex evolutionary optimization algorithms.
  arXiv  Detail & Related papers  (2025-08-16T02:00:57Z)
• Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission [87.68447072141402]
  Hybrid Language Models (HLMs) combine the low-latency efficiency of Small Language Models (SLMs) on edge devices with the high accuracy of Large Language Models (LLMs) on centralized servers.<n>We propose FedHLM, a communication-efficient HLM framework that integrates uncertainty-aware inference with Federated Learning (FL)
  arXiv  Detail & Related papers  (2025-06-30T02:56:11Z)
• Can Large Language Models Be Trusted as Evolutionary Optimizers for Network-Structured Combinatorial Problems? [8.431866560904753]
  Large Language Models (LLMs) have shown strong capabilities in language understanding and reasoning across diverse domains.<n>In this work, we propose a systematic framework to evaluate the capability of LLMs to engage with problem structures.<n>We adopt the commonly used evolutionary (EVO) and propose a comprehensive evaluation framework that rigorously assesses the output fidelity of LLM-based operators.
  arXiv  Detail & Related papers  (2025-01-25T05:19:19Z)
• In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics [0.020482269513546456]
  Large Language Models (LLMs) have shown great potential in automatically generating and optimizing (meta)heuristics. This paper presents a novel hybrid approach, LLaMEA-HPO, which integrates an open source LLaMEA framework with a Hyper- Evolutionary Optimization (HPO) procedure in the loop.
  arXiv  Detail & Related papers  (2024-10-07T14:04:31Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Large Language Models to Enhance Bayesian Optimization [57.474613739645605]
  We present LLAMBO, a novel approach that integrates the capabilities of Large Language Models (LLM) within Bayesian optimization. At a high level, we frame the BO problem in natural language, enabling LLMs to iteratively propose and evaluate promising solutions conditioned on historical evaluations. Our findings illustrate that LLAMBO is effective at zero-shot warmstarting, and enhances surrogate modeling and candidate sampling, especially in the early stages of search when observations are sparse.
  arXiv  Detail & Related papers  (2024-02-06T11:44:06Z)
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [70.38817963253034]
  This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution. We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
  arXiv  Detail & Related papers  (2023-09-01T09:40:36Z)
• A General Recipe for Likelihood-free Bayesian Optimization [115.82591413062546]
  We propose likelihood-free BO (LFBO) to extend BO to a broader class of models and utilities. LFBO directly models the acquisition function without having to separately perform inference with a probabilistic surrogate model. We show that computing the acquisition function in LFBO can be reduced to optimizing a weighted classification problem.
  arXiv  Detail & Related papers  (2022-06-27T03:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.