Large Language Bayes

• URL: http://arxiv.org/abs/2504.14025v1
• Date: Fri, 18 Apr 2025 18:30:29 GMT
• Title: Large Language Bayes
• Authors: Justin Domke,
• Abstract summary: This paper takes an informal problem description as input, and combines a large language model and a probabilistic programming language.<n>A posterior over latent variables follows by conditioning on observed data and integrating over formal models.<n>We show that this produces sensible predictions without the need to specify a formal model.
• Score: 22.372504018202154
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many domain experts do not have the time or training to write formal Bayesian models. This paper takes an informal problem description as input, and combines a large language model and a probabilistic programming language to create a joint distribution over formal models, latent variables, and data. A posterior over latent variables follows by conditioning on observed data and integrating over formal models. This presents a challenging inference problem. We suggest an inference recipe that amounts to generating many formal models from the large language model, performing approximate inference on each, and then doing a weighted average. This is justified an analyzed as a combination of self-normalized importance sampling, MCMC, and variational inference. We show that this produces sensible predictions without the need to specify a formal model.

Related papers

• From Black-box to Causal-box: Towards Building More Interpretable Models [57.23201263629627]
  We introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models.<n>We derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query.
  arXiv  Detail & Related papers  (2025-10-24T20:03:18Z)
• Do Language Models Have Bayesian Brains? Distinguishing Stochastic and Deterministic Decision Patterns within Large Language Models [2.5812117322021644]
  We show that under certain conditions, language models can exhibit near-deterministic decision-making.<n>This challenges the sampling assumption and undermines previous methods for eliciting human-like priors.<n>We propose a straightforward approach to distinguish between deterministic decision patterns in Gibbs sampling, helping to prevent the inference of misleading language model priors.
  arXiv  Detail & Related papers  (2025-06-12T01:23:22Z)
• Causal Estimation of Memorisation Profiles [58.20086589761273]
  Understanding memorisation in language models has practical and societal implications. Memorisation is the causal effect of training with an instance on the model's ability to predict that instance. This paper proposes a new, principled, and efficient method to estimate memorisation based on the difference-in-differences design from econometrics.
  arXiv  Detail & Related papers  (2024-06-06T17:59:09Z)
• BayesBlend: Easy Model Blending using Pseudo-Bayesian Model Averaging, Stacking and Hierarchical Stacking in Python [0.0]
  We introduce the BayesBlend Python package to estimate weights and blend multiple (Bayesian) models' predictive distributions. BayesBlend implements pseudo-Bayesian model averaging, stacking and, uniquely, hierarchical Bayesian stacking to estimate model weights. We demonstrate the usage of BayesBlend with examples of insurance loss modeling.
  arXiv  Detail & Related papers  (2024-04-30T19:15:33Z)
• Forcing Diffuse Distributions out of Language Models [70.28345569190388]
  Despite being trained specifically to follow user instructions, today's instructiontuned language models perform poorly when instructed to produce random outputs. We propose a fine-tuning method that encourages language models to output distributions that are diffuse over valid outcomes.
  arXiv  Detail & Related papers  (2024-04-16T19:17:23Z)
• Multilingual Mathematical Autoformalization [14.433478397963123]
  Autoformalization is the task of translating natural language materials into machine-verifiable formalisations. Existing methods tend to circumvent this challenge by manually curating small corpora. In this work, we create $textttMMA$, a large, flexible, multilingual, and multi-domain dataset of informal-formal pairs.
  arXiv  Detail & Related papers  (2023-11-07T06:42:15Z)
• PAMI: partition input and aggregate outputs for model interpretation [69.42924964776766]
  In this study, a simple yet effective visualization framework called PAMI is proposed based on the observation that deep learning models often aggregate features from local regions for model predictions. The basic idea is to mask majority of the input and use the corresponding model output as the relative contribution of the preserved input part to the original model prediction. Extensive experiments on multiple tasks confirm the proposed method performs better than existing visualization approaches in more precisely finding class-specific input regions.
  arXiv  Detail & Related papers  (2023-02-07T08:48:34Z)
• Black-box language model explanation by context length probing [7.526153863886609]
  We present context length probing, a novel explanation technique for causal language models. The technique is model-agnostic and does not rely on access to model internals beyond computing token-level probabilities. We apply context length probing to large pre-trained language models and offer some initial analyses and insights.
  arXiv  Detail & Related papers  (2022-12-30T16:24:10Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Understanding Text Classification Data and Models Using Aggregated Input Salience [2.105564340986074]
  In some cases, an input salience method, which highlights the most important parts of the input, may reveal problematic reasoning. In this paper we aim to address these issues and go from understanding single examples to understanding entire datasets and models. Using this methodology we address multiple distinct but common model developer needs by showing how problematic data and model behavior can be identified.
  arXiv  Detail & Related papers  (2022-11-10T11:00:57Z)
• Low-Rank Constraints for Fast Inference in Structured Models [110.38427965904266]
  This work demonstrates a simple approach to reduce the computational and memory complexity of a large class of structured models. Experiments with neural parameterized structured models for language modeling, polyphonic music modeling, unsupervised grammar induction, and video modeling show that our approach matches the accuracy of standard models at large state spaces.
  arXiv  Detail & Related papers  (2022-01-08T00:47:50Z)
• Generative Text Modeling through Short Run Inference [47.73892773331617]
  The present work proposes a short run dynamics for inference. It is variation from the prior distribution of the latent variable and then runs a small number of Langevin dynamics steps guided by its posterior distribution. We show that the models trained with short run dynamics more accurately model the data, compared to strong language model and VAE baselines, and exhibit no sign of posterior collapse.
  arXiv  Detail & Related papers  (2021-05-27T09:14:35Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)
• To what extent do human explanations of model behavior align with actual model behavior? [91.67905128825402]
  We investigated the extent to which human-generated explanations of models' inference decisions align with how models actually make these decisions. We defined two alignment metrics that quantify how well natural language human explanations align with model sensitivity to input words. We find that a model's alignment with human explanations is not predicted by the model's accuracy on NLI.
  arXiv  Detail & Related papers  (2020-12-24T17:40:06Z)
• A Gamma-Poisson Mixture Topic Model for Short Text [0.0]
  Most topic models are constructed under the assumption that documents follow a multinomial distribution. For topic modelling, the Poisson distribution describes the number of occurrences of a word in documents of fixed length. The few Poisson topic models in literature are admixture models, making the assumption that a document is generated from a mixture of topics.
  arXiv  Detail & Related papers  (2020-04-23T21:13:53Z)
• Limits of Detecting Text Generated by Large-Scale Language Models [65.46403462928319]
  Some consider large-scale language models that can generate long and coherent pieces of text as dangerous, since they may be used in misinformation campaigns. Here we formulate large-scale language model output detection as a hypothesis testing problem to classify text as genuine or generated.
  arXiv  Detail & Related papers  (2020-02-09T19:53:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.