Feature Generation Using LLMs: An Evolutionary Algorithm Approach

• URL: http://arxiv.org/abs/2607.16255v1
• Date: Sat, 27 Jun 2026 12:12:40 GMT
• Title: Feature Generation Using LLMs: An Evolutionary Algorithm Approach
• Abstract summary: Large Language Models have shown abilities in coding, mathematical reasoning, and processing world knowledge.<n>We have created a pipeline that takes a set of attributes and a prompt to generate new features.<n>Our results show that, in most cases, the language model can produce new features based on mathematical and logical operators.
• Score: 0.7919810878571298
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A crucial step in machine learning pipelines is to present each entity with features or attributes that are representative of the characteristics of the processed entities. Feature engineering is an important step in finding a relation among attributes that otherwise may not be processed by the ML algorithms. Meanwhile, Large Language Models have shown promising abilities in coding, mathematical reasoning, and processing world knowledge. In this work, we utilize an LLM for the problem of feature generation from tabular data based on the previously given features. We have created a pipeline that takes a set of attributes and a prompt to generate new features. Then, our selection algorithm selects the best-performing sets of attributes. We apply our method to eight datasets from different domains and data types. Our results show that, in most cases, the language model can produce new features based on mathematical and logical operators that are useful for the given tasks and can improve classification results.

Related papers

• Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data [52.80567461420316]
  We propose a Memory-Augmented LLM-based Multi-Agent System (textbfMALMAS) for automated feature generation.<n>MALMAS decomposes the generation process into agents with distinct responsibilities, and a Router Agent activates an appropriate subset of agents per iteration.<n>We further integrate a memory module comprising procedural memory, feedback memory, and conceptual memory, enabling iterative refinement that adaptively guides subsequent feature generation.
  arXiv  Detail & Related papers  (2026-04-22T07:09:30Z)
• Neuron-Aware Data Selection In Instruction Tuning For Large Language Models [69.08560711834848]
  Instruction Tuning (IT) has been proven to be an effective approach to unlock the powerful capabilities of large language models (LLMs)<n>Recent studies indicate that excessive IT data can degrade LLMs performance, while carefully selecting a small subset of high-quality IT data can significantly enhance their capabilities.<n>We propose a novel and efficient framework called NAIT to identify the most efficient subset data from the IT dataset.
  arXiv  Detail & Related papers  (2026-03-13T17:39:03Z)
• Tabular Feature Discovery With Reasoning Type Exploration [5.030210915367596]
  Large language models (LLMs) have been used to automatically generate new features by leveraging their vast knowledge.<n>We propose a novel method REFeat, which guides an LLM to discover diverse and informative features by leveraging multiple types of reasoning.
  arXiv  Detail & Related papers  (2025-06-25T12:18:34Z)
• Re-Initialization Token Learning for Tool-Augmented Large Language Models [49.91503552002649]
  Large language models have demonstrated exceptional performance, yet struggle with complex tasks such as numerical reasoning, plan generation.<n>We propose a novel token learning method that aligns tool tokens with the existing word embedding space.<n>We evaluate the method on tasks such as numerical reasoning, knowledge-based question answering, and embodied plan generation using GSM8K-XL, FuncQA, KAMEL, and VirtualHome datasets.
  arXiv  Detail & Related papers  (2025-06-17T07:11:00Z)
• Embedding Domain-Specific Knowledge from LLMs into the Feature Engineering Pipeline [0.0]
  We propose using Large Language Models (LLMs) as an initial feature construction step to add knowledge to the dataset.<n>Our results show that the evolution can converge faster, saving us computational resources.
  arXiv  Detail & Related papers  (2025-03-27T04:48:58Z)
• CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection [30.286598189230396]
  Class-Agnostic Shared Attribute Base encourages the model to learn category-agnostic attributes shared across incremental classes.<n>For subsequent tasks, the retained attributes are frozen, and new attributes are selected from the remaining candidates.<n>Experiments on the COCO dataset demonstrate the state-of-the-art performance of our method.
  arXiv  Detail & Related papers  (2024-10-08T08:36:12Z)
• LLM-Select: Feature Selection with Large Language Models [64.5099482021597]
  Large language models (LLMs) are capable of selecting the most predictive features, with performance rivaling the standard tools of data science.<n>Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place.
  arXiv  Detail & Related papers  (2024-07-02T22:23:40Z)
• Learning to Extract Structured Entities Using Language Models [52.281701191329]
  Recent advances in machine learning have significantly impacted the field of information extraction. We reformulate the task to be entity-centric, enabling the use of diverse metrics. We contribute to the field by introducing Structured Entity Extraction and proposing the Approximate Entity Set OverlaP metric.
  arXiv  Detail & Related papers  (2024-02-06T22:15:09Z)
• Learning Concise and Descriptive Attributes for Visual Recognition [25.142065847381758]
  We show that querying thousands of attributes can achieve performance competitive with image features. We propose a novel learning-to-search method to discover those concise sets of attributes.
  arXiv  Detail & Related papers  (2023-08-07T16:00:22Z)
• Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias [92.41919689753051]
  Large language models (LLMs) have been recently leveraged as training data generators for various natural language processing (NLP) tasks. We investigate training data generation with diversely attributed prompts, which have the potential to yield diverse and attributed generated data. We show that attributed prompts outperform simple class-conditional prompts in terms of the resulting model's performance.
  arXiv  Detail & Related papers  (2023-06-28T03:31:31Z)
• Explainable Model-specific Algorithm Selection for Multi-Label Classification [6.442438468509492]
  Multi-label classification (MLC) is an ML task of predictive modeling in which a data instance can simultaneously belong to multiple classes. Several MLC algorithms have been proposed in the literature, resulting in a meta-optimization problem. We investigate in this work the quality of an automated approach that uses characteristics of the datasets.
  arXiv  Detail & Related papers  (2022-11-21T07:42:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.