LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression

• URL: http://arxiv.org/abs/2505.18602v2
• Date: Fri, 08 Aug 2025 06:50:37 GMT
• Title: LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression
• Abstract summary: Large language models (LLMs) have revolutionized algorithm development, yet their application in symbolic regression remains limited.<n>We propose a meta learning framework that enables LLMs to automatically design selection operators for evolutionary symbolic regression algorithms.
• Score: 11.760077969729055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have revolutionized algorithm development, yet their application in symbolic regression, where algorithms automatically discover symbolic expressions from data, remains constrained and is typically designed manually by human experts. In this paper, we propose a meta learning framework that enables LLMs to automatically design selection operators for evolutionary symbolic regression algorithms. We first identify two key limitations in existing LLM-based algorithm evolution techniques: a lack of semantic guidance and code bloat. The absence of semantic awareness can lead to ineffective exchange of useful code components, and bloat results in unnecessarily complex components, both of which can reduce the interpretability of the designed algorithm or hinder evolutionary learning progress. To address these issues, we enhance the LLM-based evolution framework for meta symbolic regression with two key innovations: a complementary, semantics-aware selection operator and bloat control. Additionally, we embed domain knowledge into the prompt, enabling the LLM to generate more effective and contextually relevant selection operators. Our experimental results on symbolic regression benchmarks show that LLMs can devise selection operators that outperform nine expert-designed baselines, achieving state-of-the-art performance. Moreover, the evolved operator can further improve the state-of-the-art symbolic regression algorithm, achieving the best performance among 26 symbolic regression and machine learning algorithms across 116 regression datasets. This demonstrates that LLMs can exceed expert-level algorithm design for symbolic regression.

Related papers

• From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities [103.60384544335072]
  Large Language Models (LLMs) are evolving from performing end-to-end mathematical reasoning to integrating agentic intelligence.<n>Our framework aligns problem-solving agentic behaviors with a structured taxonomy of reusable mathematical atomic capabilities.<n> Experiments reveal that models with similar end-to-end accuracy can exhibit markedly different agentic capability profiles.
  arXiv  Detail & Related papers  (2026-08-27T10:52:06Z)
• FormuEvo: LLM-Guided Evolution for Discovering Solver-Efficient Mixed-Integer Programming Formulations [66.32852384262038]
  Mixed-integer programming (MIP) lies at the core of operations research and industrial optimization.<n>We propose FormuEvo, an evolutionary framework for automated discovery of solver-efficient MIP formulations.<n>FormuEvo frames MIP formulation design as evolutionary optimization over the symbolic space of MIP formulations.
  arXiv  Detail & Related papers  (2026-08-24T15:01:28Z)
• Deliberate Evolution: Agentic Reasoning for Sample-Efficient Symbolic Regression with LLMs [52.89778838903305]
  Deliberate Evolution (DE) is an agentic framework that decouples symbolic generation from search control.<n>Experiments on LLM-SRBench show that DE consistently outperforms representative LLM-based SR baselines across diverse scientific domains.
  arXiv  Detail & Related papers  (2026-06-03T02:22:16Z)
• Programmatic Context Augmentation for LLM-based Symbolic Regression [65.01826333382738]
  Large language model (LLM)-based evolutionary search methods have been introduced into Symbolic regression.<n>We propose a novel LLM-based evolutionary search framework that incorporates programmatic context augmentation.<n>By enabling code-based interactions with the dataset, our method can actively perform data analysis and extract informative signals.
  arXiv  Detail & Related papers  (2026-05-04T19:34:03Z)
• LLaMEA-SAGE: Guiding Automated Algorithm Design with Structural Feedback from Explainable AI [4.440668887299803]
  Large language models have enabled automated algorithm design (AAD) by generating optimization algorithms directly from natural-language prompts.<n>We propose a mechanism for guiding AAD using feedback constructed from graph-theoretic and complexity features extracted from the abstract syntax trees of the generated algorithms.<n>We show that the proposed structured guidance achieves the same performance faster than vanilla LLaMEA in a small controlled experiment.
  arXiv  Detail & Related papers  (2026-01-29T10:27:29Z)
• Evolutionary Strategies lead to Catastrophic Forgetting in LLMs [51.91763220981834]
  Evolutionary Strategies (ES) have recently re-emerged as a gradient-free alternative to traditional learning algorithms.<n>ES is able to reach performance numbers close to GRPO for math and reasoning tasks with a comparable compute budget.<n>ES is accompanied by significant forgetting of prior abilities, limiting its applicability for training models online.
  arXiv  Detail & Related papers  (2026-01-28T18:59:34Z)
• Experience-Guided Reflective Co-Evolution of Prompts and Heuristics for Automatic Algorithm Design [124.54166764570972]
  Combinatorial optimization problems are traditionally tackled with handcrafted algorithms.<n>Recent progress has highlighted the potential of automatics design powered by large language models.<n>We propose the Experience-Evolution Reflective Co-Guided of Prompt and Heuristics (EvoPH) for automatic algorithm design.
  arXiv  Detail & Related papers  (2025-09-29T09:24:09Z)
• Teaching LLMs to Think Mathematically: A Critical Study of Decision-Making via Optimization [1.246870021158888]
  This paper investigates the capabilities of large language models (LLMs) in formulating and solving decision-making problems using mathematical programming.<n>We first conduct a systematic review and meta-analysis of recent literature to assess how well LLMs understand, structure, and solve optimization problems across domains.<n>Our systematic evidence is complemented by targeted experiments designed to evaluate the performance of state-of-the-art LLMs in automatically generating optimization models for problems in computer networks.
  arXiv  Detail & Related papers  (2025-08-25T14:52:56Z)
• Agentic Reinforced Policy Optimization [66.96989268893932]
  Large-scale reinforcement learning with verifiable rewards (RLVR) has demonstrated its effectiveness in harnessing the potential of large language models (LLMs) for single-turn reasoning tasks.<n>Current RL algorithms inadequately balance the models' intrinsic long-horizon reasoning capabilities and their proficiency in multi-turn tool interactions.<n>We propose Agentic Reinforced Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training multi-turn LLM-based agents.
  arXiv  Detail & Related papers  (2025-07-26T07:53:11Z)
• Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey [69.45421620616486]
  This work presents the first structured taxonomy and analysis of discrete tokenization methods designed for large language models (LLMs)<n>We categorize 8 representative VQ variants that span classical and modern paradigms and analyze their algorithmic principles, training dynamics, and integration challenges with LLM pipelines.<n>We identify key challenges including codebook collapse, unstable gradient estimation, and modality-specific encoding constraints.
  arXiv  Detail & Related papers  (2025-07-21T10:52:14Z)
• SAFT: Structure-Aware Fine-Tuning of LLMs for AMR-to-Text Generation [50.277959544420455]
  SAFT is a structure-aware fine-tuning approach that injects graph topology into pretrained language models.<n>We compute direction-sensitive positional encodings from the magnetic Laplacian of transformed AMRs.<n> SAFT sets a new state-of-the-art on AMR 3.0 with a 3.5 BLEU improvement over baselines.
  arXiv  Detail & Related papers  (2025-07-15T18:12:57Z)
• Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery [2.0958996366233094]
  We investigate meta-heuristic optimisation algorithms automatically generated by Large Language Model driven algorithm discovery methods.<n>We iteratively evolve black-box optimisations, evaluated on 10 functions from the BBOB benchmark suite.<n>We log behavioural metrics including exploration, exploitation, convergence and stagnation measures, for each run, and analyse these via visual projections and network-based representations.
  arXiv  Detail & Related papers  (2025-07-04T14:19:39Z)
• Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning [50.53703102032562]
  Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks.<n>The mechanism behind this capability, known as in-context learning (ICL), remains both controversial and poorly understood.
  arXiv  Detail & Related papers  (2025-05-16T08:50:42Z)
• LLM-Guided Evolution: An Autonomous Model Optimization for Object Detection [0.0]
  In machine learning, Neural Architecture Search (NAS) requires domain knowledge of model design and a large amount of trial-and-error to achieve promising performance.<n>The Large Language Model (LLM)-Guided Evolution (GE) framework transformed this approach by incorporating LLMs to directly modify model source code for image classification algorithms on CIFAR data.<n>We show that LLM-GE produced variants with significant performance improvements, such as an increase in Mean Average Precision from 92.5% to 94.5%.
  arXiv  Detail & Related papers  (2025-04-03T05:06:06Z)
• LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers [10.282327560070202]
  Large Language Models (LLMs) have enabled the integration of domain knowledge into the feature engineering process.<n>We propose LLM-FE, a novel framework that combines evolutionary search with the domain knowledge and reasoning capabilities of LLMs.<n>Our results demonstrate that LLM-FE consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-03-18T17:11:24Z)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning [52.39395405893965]
  Large language models (LLMs) have demonstrated remarkable capabilities across a range of tasks.<n>However, they still struggle with problems requiring multi-step decision-making and environmental feedback.<n>We propose a framework that can automatically learn a reward model from the environment without human annotations.
  arXiv  Detail & Related papers  (2025-02-17T18:49:25Z)
• Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up [9.42385235462794]
  Large language models (LLMs) have shown remarkable performance in reasoning tasks but face limitations in mathematical and complex logical reasoning.<n>We propose Reversal of Thought (RoT) to enhance the logical reasoning abilities of LLMs during the warm-up phase prior to batch inference.<n>RoT utilizes a Preference-Guided Reverse Reasoning warm-up strategy, which integrates logical symbols for pseudocode planning.
  arXiv  Detail & Related papers  (2024-10-16T07:44:28Z)
• Cognitive LLMs: Towards Integrating Cognitive Architectures and Large Language Models for Manufacturing Decision-making [51.737762570776006]
  LLM-ACTR is a novel neuro-symbolic architecture that provides human-aligned and versatile decision-making. Our framework extracts and embeds knowledge of ACT-R's internal decision-making process as latent neural representations. Our experiments on novel Design for Manufacturing tasks show both improved task performance as well as improved grounded decision-making capability.
  arXiv  Detail & Related papers  (2024-08-17T11:49:53Z)
• CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models [68.64605538559312]
  In this paper, we analyze the MLLM instruction tuning from both theoretical and empirical perspectives. Inspired by our findings, we propose a measurement to quantitatively evaluate the learning balance. In addition, we introduce an auxiliary loss regularization method to promote updating of the generation distribution of MLLMs.
  arXiv  Detail & Related papers  (2024-07-29T23:18:55Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs)<n>We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• Large Language Models As Evolution Strategies [6.873777465945062]
  In this work, we investigate whether large language models (LLMs) are in principle capable of implementing evolutionary optimization algorithms. We introduce a novel prompting strategy, consisting of least-to-most sorting of discretized population members. We find that our setup allows the user to obtain an LLM-based evolution strategy, which we call EvoLLM', that robustly outperforms baseline algorithms.
  arXiv  Detail & Related papers  (2024-02-28T15:02:17Z)
• How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
  Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities. We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
  arXiv  Detail & Related papers  (2024-02-25T20:07:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.