LLM-BI: Towards Fully Automated Bayesian Inference with Large Language Models

• URL: http://arxiv.org/abs/2508.08300v1
• Date: Thu, 07 Aug 2025 00:00:59 GMT
• Title: LLM-BI: Towards Fully Automated Bayesian Inference with Large Language Models
• Authors: Yongchao Huang,
• Abstract summary: This paper investigates the feasibility of using a Large Language Model (LLM) to automate the specification of prior distributions and likelihoods.<n>As a proof-of-concept, we present two experiments focused on Bayesian linear regression.<n>Our results validate the potential of LLMs to automate key steps in Bayesian modeling.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A significant barrier to the widespread adoption of Bayesian inference is the specification of prior distributions and likelihoods, which often requires specialized statistical expertise. This paper investigates the feasibility of using a Large Language Model (LLM) to automate this process. We introduce LLM-BI (Large Language Model-driven Bayesian Inference), a conceptual pipeline for automating Bayesian workflows. As a proof-of-concept, we present two experiments focused on Bayesian linear regression. In Experiment I, we demonstrate that an LLM can successfully elicit prior distributions from natural language. In Experiment II, we show that an LLM can specify the entire model structure, including both priors and the likelihood, from a single high-level problem description. Our results validate the potential of LLMs to automate key steps in Bayesian modeling, enabling the possibility of an automated inference pipeline for probabilistic programming.

Related papers

• Utilizing Large Language Models for Machine Learning Explainability [37.31918138232927]
  This study explores the explainability capabilities of large language models (LLMs), when employed to autonomously generate machine learning (ML) solutions.<n>Three state-of-the-art LLMs are prompted to design training pipelines for four common classifiers: Random Forest, XGBoost, Multilayer Perceptron, and Long Short-Term Memory networks.<n>The generated models are evaluated in terms of predictive performance (recall, precision, and F1-score) and explainability using SHAP (SHapley Additive exPlanations)
  arXiv  Detail & Related papers  (2025-10-08T11:46:23Z)
• Can Linear Probes Measure LLM Uncertainty? [0.0]
  Uncertainty Quantification (UQ) represents a key aspect for reliable deployment of Large Language Models (LLMs) in automated decision-making and beyond.<n>We show that taking a principled approach via Bayesian statistics leads to improved performance despite leveraging the simplest possible model, namely linear regression.<n>We infer the global uncertainty level of the LLM by identifying a sparse combination of distributional features, leading to an efficient UQ scheme.
  arXiv  Detail & Related papers  (2025-10-05T09:14:57Z)
• Probabilistic Token Alignment for Large Language Model Fusion [100.30692772017238]
  Training large language models (LLMs) from scratch can yield models with unique functionalities and strengths, but it is costly and often leads to redundant capabilities.<n>A key challenge in existing model fusion is their dependence on manually predefined vocabulary alignment.<n>We propose the probabilistic token alignment method as a general and soft mapping for alignment, named as PTA-LLM.
  arXiv  Detail & Related papers  (2025-09-21T23:18:24Z)
• BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design [20.03498575187842]
  We propose a general-purpose approach for improving the ability of Large Language Models (LLMs) to intelligently and adaptively gather information from a user or other external source.<n>Our approach, which we call BED-LLM, is based on iteratively choosing questions that maximize the expected information gain.<n>We find that BED-LLM achieves substantial gains in performance across a range of tests based on the 20 questions game.
  arXiv  Detail & Related papers  (2025-08-28T19:51:43Z)
• Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization [22.286144400569007]
  We evaluate the potential of Large Language Models (LLMs) in building Bayesian Networks (BNs) by approximating domain expert priors.<n>Our experiments on eighty publicly available Bayesian Networks, from healthcare to finance, demonstrate that querying LLMs about the conditional probabilities of events provides meaningful results.
  arXiv  Detail & Related papers  (2025-05-21T18:15:05Z)
• Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models [50.16340812031201]
  We show that large language models (LLMs) do not update their beliefs as expected from the Bayesian framework.<n>We teach the LLMs to reason in a Bayesian manner by training them to mimic the predictions of an optimal Bayesian model.
  arXiv  Detail & Related papers  (2025-03-21T20:13:04Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems [59.40480894948944]
  Large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. Under this model, the LLM Planner navigates a partially observable Markov decision process (POMDP) by iteratively generating language-based subgoals via prompting. We prove that the pretrained LLM Planner effectively performs Bayesian aggregated imitation learning (BAIL) through in-context learning.
  arXiv  Detail & Related papers  (2024-05-30T09:42:54Z)
• LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language [35.84181171987974]
  Our goal is to build a regression model that can process numerical data and make probabilistic predictions at arbitrary locations.<n>We start by exploring strategies for eliciting explicit, coherent numerical predictive distributions from Large Language Models.<n>We demonstrate the ability to usefully incorporate text into numerical predictions, improving predictive performance and giving quantitative structure that reflects qualitative descriptions.
  arXiv  Detail & Related papers  (2024-05-21T15:13:12Z)
• Amortizing intractable inference in large language models [56.92471123778389]
  We use amortized Bayesian inference to sample from intractable posterior distributions. We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem.
  arXiv  Detail & Related papers  (2023-10-06T16:36:08Z)
• ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models [69.50316788263433]
  We propose ProbVLM, a probabilistic adapter that estimates probability distributions for the embeddings of pre-trained vision-language models. We quantify the calibration of embedding uncertainties in retrieval tasks and show that ProbVLM outperforms other methods. We present a novel technique for visualizing the embedding distributions using a large-scale pre-trained latent diffusion model.
  arXiv  Detail & Related papers  (2023-07-01T18:16:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.