LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language

• URL: http://arxiv.org/abs/2405.12856v4
• Date: Tue, 22 Oct 2024 19:53:58 GMT
• Title: LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language
• Authors: James Requeima, John Bronskill, Dami Choi, Richard E. Turner, David Duvenaud,
• Abstract summary: Our goal is to build a regression model that can process numerical data and make probabilistic predictions at arbitrary locations. We start by exploring strategies for eliciting explicit, coherent numerical predictive distributions from Large Language Models. We demonstrate the ability to usefully incorporate text into numerical predictions, improving predictive performance and giving quantitative structure that reflects qualitative descriptions.
• Score: 35.84181171987974
• License:
• Abstract: Machine learning practitioners often face significant challenges in formally integrating their prior knowledge and beliefs into predictive models, limiting the potential for nuanced and context-aware analyses. Moreover, the expertise needed to integrate this prior knowledge into probabilistic modeling typically limits the application of these models to specialists. Our goal is to build a regression model that can process numerical data and make probabilistic predictions at arbitrary locations, guided by natural language text which describes a user's prior knowledge. Large Language Models (LLMs) provide a useful starting point for designing such a tool since they 1) provide an interface where users can incorporate expert insights in natural language and 2) provide an opportunity for leveraging latent problem-relevant knowledge encoded in LLMs that users may not have themselves. We start by exploring strategies for eliciting explicit, coherent numerical predictive distributions from LLMs. We examine these joint predictive distributions, which we call LLM Processes, over arbitrarily-many quantities in settings such as forecasting, multi-dimensional regression, black-box optimization, and image modeling. We investigate the practical details of prompting to elicit coherent predictive distributions, and demonstrate their effectiveness at regression. Finally, we demonstrate the ability to usefully incorporate text into numerical predictions, improving predictive performance and giving quantitative structure that reflects qualitative descriptions. This lets us begin to explore the rich, grounded hypothesis space that LLMs implicitly encode.

Related papers

• Using Large Language Models for Expert Prior Elicitation in Predictive Modelling [53.54623137152208]
  This study proposes using large language models (LLMs) to elicit expert prior distributions for predictive models. We compare LLM-elicited and uninformative priors, evaluate whether LLMs truthfully generate parameter distributions, and propose a model selection strategy for in-context learning and prior elicitation. Our findings show that LLM-elicited prior parameter distributions significantly reduce predictive error compared to uninformative priors in low-data settings.
  arXiv  Detail & Related papers  (2024-11-26T10:13:39Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a time series forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context. We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters. Our experiments highlight the importance of incorporating contextual information, demonstrate surprising performance when using LLM-based forecasting models, and also reveal some of their critical shortcomings.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• A Law of Next-Token Prediction in Large Language Models [30.265295018979078]
  We introduce a precise and quantitative law that governs the learning of contextualized token embeddings through intermediate layers in pre-trained large language models. Our findings reveal that each layer contributes equally to enhancing prediction accuracy, from the lowest to the highest layer.
  arXiv  Detail & Related papers  (2024-08-24T02:48:40Z)
• Translating Expert Intuition into Quantifiable Features: Encode Investigator Domain Knowledge via LLM for Enhanced Predictive Analytics [2.330270848695646]
  This paper explores the potential of Large Language Models to bridge the gap by systematically converting investigator-derived insights into quantifiable, actionable features. We present a framework that leverages LLMs' natural language understanding capabilities to encode these red flags into a structured feature set that can be readily integrated into existing predictive models. The results indicate significant improvements in risk assessment and decision-making accuracy, highlighting the value of blending human experiential knowledge with advanced machine learning techniques.
  arXiv  Detail & Related papers  (2024-05-11T13:23:43Z)
• Understanding Privacy Risks of Embeddings Induced by Large Language Models [75.96257812857554]
  Large language models show early signs of artificial general intelligence but struggle with hallucinations. One promising solution is to store external knowledge as embeddings, aiding LLMs in retrieval-augmented generation. Recent studies experimentally showed that the original text can be partially reconstructed from text embeddings by pre-trained language models.
  arXiv  Detail & Related papers  (2024-04-25T13:10:48Z)
• Quantitative knowledge retrieval from large language models [4.155711233354597]
  Large language models (LLMs) have been extensively studied for their abilities to generate convincing natural language sequences. This paper explores the feasibility of LLMs as a mechanism for quantitative knowledge retrieval to aid data analysis tasks.
  arXiv  Detail & Related papers  (2024-02-12T16:32:37Z)
• Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models [54.21695754082441]
  We propose a framework to teach Large Language Models (LLMs) to generate explainable stock predictions. A reflective agent learns how to explain past stock movements through self-reasoning, while the PPO trainer trains the model to generate the most likely explanations. Our framework can outperform both traditional deep-learning and LLM methods in prediction accuracy and Matthews correlation coefficient.
  arXiv  Detail & Related papers  (2024-02-06T03:18:58Z)
• Evaluating and Explaining Large Language Models for Code Using Syntactic Structures [74.93762031957883]
  This paper introduces ASTxplainer, an explainability method specific to Large Language Models for code. At its core, ASTxplainer provides an automated method for aligning token predictions with AST nodes. We perform an empirical evaluation on 12 popular LLMs for code using a curated dataset of the most popular GitHub projects.
  arXiv  Detail & Related papers  (2023-08-07T18:50:57Z)
• Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models [51.3422222472898]
  We document the capability of large language models (LLMs) like ChatGPT to predict stock price movements using news headlines. We develop a theoretical model incorporating information capacity constraints, underreaction, limits-to-arbitrage, and LLMs.
  arXiv  Detail & Related papers  (2023-04-15T19:22:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.