In Context Learning and Reasoning for Symbolic Regression with Large Language Models

• URL: http://arxiv.org/abs/2410.17448v1
• Date: Tue, 22 Oct 2024 21:50:52 GMT
• Title: In Context Learning and Reasoning for Symbolic Regression with Large Language Models
• Authors: Samiha Sharlin, Tyler R. Josephson,
• Abstract summary: Large Language Models (LLMs) are transformer-based machine learning models. We show how GPT-4 can perform symbolic regression on equations from datasets. This approach does not outperform established SR programs where target equations are more complex.
• Score: 0.0
• License:
• Abstract: Large Language Models (LLMs) are transformer-based machine learning models that have shown remarkable performance in tasks for which they were not explicitly trained. Here, we explore the potential of LLMs to perform symbolic regression -- a machine-learning method for finding simple and accurate equations from datasets. We prompt GPT-4 to suggest expressions from data, which are then optimized and evaluated using external Python tools. These results are fed back to GPT-4, which proposes improved expressions while optimizing for complexity and loss. Using chain-of-thought prompting, we instruct GPT-4 to analyze the data, prior expressions, and the scientific context (expressed in natural language) for each problem before generating new expressions. We evaluated the workflow in rediscovery of five well-known scientific equations from experimental data, and on an additional dataset without a known equation. GPT-4 successfully rediscovered all five equations, and in general, performed better when prompted to use a scratchpad and consider scientific context. We also demonstrate how strategic prompting improves the model's performance and how the natural language interface simplifies integrating theory with data. Although this approach does not outperform established SR programs where target equations are more complex, LLMs can nonetheless iterate toward improved solutions while following instructions and incorporating scientific context in natural language.

Related papers

• ReasonAgain: Using Extractable Symbolic Programs to Evaluate Mathematical Reasoning [54.70811660561151]
  Existing math datasets evaluate the reasoning abilities of large language models (LLMs) by either using the final answer or the intermediate reasoning steps derived from static examples. We seek to use symbolic programs as a means for automated evaluation if a model can consistently produce correct final answers across various inputs to the program. We observe significant accuracy drops using our proposed evaluation compared with original static examples, suggesting the fragility of math reasoning in state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2024-10-24T18:02:37Z)
• OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling [62.19438812624467]
  Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning. We propose OptiBench, a benchmark for End-to-end optimization problem-solving with human-readable inputs and outputs.
  arXiv  Detail & Related papers  (2024-07-13T13:27:57Z)
• MLLM-SR: Conversational Symbolic Regression base Multi-Modal Large Language Models [13.136507215114722]
  MLLM-SR is a conversational symbolic regression method that can generate expressions that meet the requirements simply by describing the requirements with natural language instructions. We experimentally demonstrate that MLLM-SR can well understand the prior knowledge we add to the natural language instructions.
  arXiv  Detail & Related papers  (2024-06-08T09:17:54Z)
• JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models [110.45794710162241]
  Existing work either collects large-scale math-related texts for pre-training, or relies on stronger LLMs to synthesize massive math problems. We propose an efficient way that trains a small LLM for math problem synthesis, to efficiently generate sufficient high-quality pre-training data. We leverage it to synthesize 6 million math problems for pre-training our JiuZhang3.0 model, which only needs to invoke GPT-4 API 9.3k times and pre-train on 4.6B data.
  arXiv  Detail & Related papers  (2024-05-23T09:43:19Z)
• LLM4ED: Large Language Models for Automatic Equation Discovery [0.8644909837301149]
  We introduce a new framework that utilizes natural language-based prompts to guide large language models in automatically mining governing equations from data. Specifically, we first utilize the generation capability of LLMs to generate diverse equations in string form, and then evaluate the generated equations based on observations. Experiments are extensively conducted on both partial differential equations and ordinary differential equations.
  arXiv  Detail & Related papers  (2024-05-13T14:03:49Z)
• MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents [62.02920842630234]
  We show how to build small fact-checking models that have GPT-4-level performance but for 400x lower cost. We do this by constructing synthetic training data with GPT-4, which involves creating realistic yet challenging instances of factual errors. For evaluation, we unify datasets from recent work on fact-checking and grounding LLM generations into a new benchmark, LLM-AggreFact.
  arXiv  Detail & Related papers  (2024-04-16T17:59:10Z)
• Generative Pre-Trained Transformer for Symbolic Regression Base In-Context Reinforcement Learning [12.660401635672967]
  Finding mathematical formulas from observational data is a major demand of scientific research. FormulaGPT achieves the state-of-the-art performance in fitting ability compared with four baselines.
  arXiv  Detail & Related papers  (2024-04-09T14:08:47Z)
• Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator [63.762209407570715]
  Genixer is a comprehensive data generation pipeline consisting of four key steps. A synthetic VQA-like dataset trained with LLaVA1.5 enhances performance on 10 out of 12 multimodal benchmarks. MLLMs trained with task-specific datasets can surpass GPT-4V in generating complex instruction tuning data.
  arXiv  Detail & Related papers  (2023-12-11T09:44:41Z)
• Visual Instruction Tuning [79.70923292053097]
  We present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%.
  arXiv  Detail & Related papers  (2023-04-17T17:59:25Z)
• Deep Generative Symbolic Regression with Monte-Carlo-Tree-Search [29.392036559507755]
  Symbolic regression is a problem of learning a symbolic expression from numerical data. Deep neural models trained on procedurally-generated synthetic datasets showed competitive performance. We propose a novel method which provides the best of both worlds, based on a Monte-Carlo Tree Search procedure.
  arXiv  Detail & Related papers  (2023-02-22T09:10:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.