In-Context Symbolic Regression: Leveraging Large Language Models for Function Discovery

• URL: http://arxiv.org/abs/2404.19094v2
• Date: Wed, 17 Jul 2024 15:29:18 GMT
• Title: In-Context Symbolic Regression: Leveraging Large Language Models for Function Discovery
• Authors: Matteo Merler, Katsiaryna Haitsiukevich, Nicola Dainese, Pekka Marttinen,
• Abstract summary: In this work, we introduce the first comprehensive framework that utilizes Large Language Models (LLMs) for the task of Symbolic Regression. We propose In-Context Symbolic Regression (ICSR), an SR method which iteratively refines a functional form with an external LLM and determines its coefficients with an external LLM. Our findings reveal that LLMs are able to successfully find symbolic equations that fit the given data, matching or outperforming the overall performance of the best SR baselines on four popular benchmarks.
• Score: 5.2387832710686695
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State of the art Symbolic Regression (SR) methods currently build specialized models, while the application of Large Language Models (LLMs) remains largely unexplored. In this work, we introduce the first comprehensive framework that utilizes LLMs for the task of SR. We propose In-Context Symbolic Regression (ICSR), an SR method which iteratively refines a functional form with an LLM and determines its coefficients with an external optimizer. ICSR leverages LLMs' strong mathematical prior both to propose an initial set of possible functions given the observations and to refine them based on their errors. Our findings reveal that LLMs are able to successfully find symbolic equations that fit the given data, matching or outperforming the overall performance of the best SR baselines on four popular benchmarks, while yielding simpler equations with better out of distribution generalization.

Related papers

• Understanding LLM Embeddings for Regression [8.095573259696092]
  This paper provides one of the first comprehensive investigations into embedding-based regression. We show that LLM embeddings as features can be better for high-dimensional regression tasks than using traditional feature engineering. We quantify the contribution of different model effects, most notably model size and language understanding, which we find surprisingly do not always improve regression performance.
  arXiv  Detail & Related papers  (2024-11-22T03:33:51Z)
• Towards Scalable Semantic Representation for Recommendation [65.06144407288127]
  Mixture-of-Codes is proposed to construct semantic IDs based on large language models (LLMs) Our method achieves superior discriminability and dimension robustness scalability, leading to the best scale-up performance in recommendations.
  arXiv  Detail & Related papers  (2024-10-12T15:10:56Z)
• Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Prompting [21.04933334040135]
  We introduce the Self-Prompting framework, a novel method designed to fully harness the embedded RE knowledge within Large Language Models. Our framework employs a three-stage diversity approach to prompt LLMs, generating multiple synthetic samples that encapsulate specific relations from scratch. Experimental evaluations on benchmark datasets show our approach outperforms existing LLM-based zero-shot RE methods.
  arXiv  Detail & Related papers  (2024-10-02T01:12:54Z)
• Symbolic Regression with a Learned Concept Library [9.395222766576342]
  We present a novel method for searching for compact programmatic hypotheses that best explain a dataset. Our algorithm, called LaSR, uses zero-shot queries to a large language model to discover and evolve concepts. LaSR substantially outperforms a variety of state-of-the-art SR approaches based on deep learning and evolutionary algorithms.
  arXiv  Detail & Related papers  (2024-09-14T08:17:30Z)
• Empowering Few-Shot Relation Extraction with The Integration of Traditional RE Methods and Large Language Models [48.846159555253834]
  Few-Shot Relation Extraction (FSRE) appeals to more researchers in Natural Language Processing (NLP) Recent emergence of Large Language Models (LLMs) has prompted numerous researchers to explore FSRE through In-Context Learning (ICL)
  arXiv  Detail & Related papers  (2024-07-12T03:31:11Z)
• DELRec: Distilling Sequential Pattern to Enhance LLM-based Recommendation [3.5113201254928117]
  Sequential recommendation (SR) tasks enhance recommendation accuracy by capturing the connection between users' past interactions and their changing preferences. Conventional models often focus solely on capturing sequential patterns within the training data, neglecting the broader context and semantic information embedded in item titles from external sources. DelRec aims to extract knowledge from SR models and enable LLMs to easily comprehend and utilize this supplementary information for more effective sequential recommendations.
  arXiv  Detail & Related papers  (2024-06-17T02:47:09Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs) We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• Regression-aware Inference with LLMs [52.764328080398805]
  We show that an inference strategy can be sub-optimal for common regression and scoring evaluation metrics. We propose alternate inference strategies that estimate the Bayes-optimal solution for regression and scoring metrics in closed-form from sampled responses.
  arXiv  Detail & Related papers  (2024-03-07T03:24:34Z)
• Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal [49.24054920683246]
  Large language models (LLMs) suffer from catastrophic forgetting during continual learning. We propose a framework called Self-Synthesized Rehearsal (SSR) that uses the LLM to generate synthetic instances for rehearsal.
  arXiv  Detail & Related papers  (2024-03-02T16:11:23Z)
• On Learning to Summarize with Large Language Models as References [101.79795027550959]
  Large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. We study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved.
  arXiv  Detail & Related papers  (2023-05-23T16:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.