What do Language Model Probabilities Represent? From Distribution Estimation to Response Prediction

• URL: http://arxiv.org/abs/2505.02072v1
• Date: Sun, 04 May 2025 11:46:48 GMT
• Title: What do Language Model Probabilities Represent? From Distribution Estimation to Response Prediction
• Authors: Eitan Wagner, Omri Abend,
• Abstract summary: We argue that different settings lead to three distinct intended output distributions.<n>We demonstrate that NLP works often assume that these distributions should be similar, which leads to misinterpretations of their experimental findings.
• Score: 16.63148156570219
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The notion of language modeling has gradually shifted in recent years from a distribution over finite-length strings to general-purpose prediction models for textual inputs and outputs, following appropriate alignment phases. This paper analyzes the distinction between distribution estimation and response prediction in the context of LLMs, and their often conflicting goals. We examine the training phases of LLMs, which include pretraining, in-context learning, and preference tuning, and also the common use cases for their output probabilities, which include completion probabilities and explicit probabilities as output. We argue that the different settings lead to three distinct intended output distributions. We demonstrate that NLP works often assume that these distributions should be similar, which leads to misinterpretations of their experimental findings. Our work sets firmer formal foundations for the interpretation of LLMs, which will inform ongoing work on the interpretation and use of LLMs' induced distributions.

Related papers

• Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training [57.03005244917803]
  Large language models (LLMs) often fail on out-of-distribution (OOD) samples due to spurious correlations acquired during pre-training.<n>Here, we aim to mitigate such spurious correlations through causality-aware post-training (CAPT)<n> Experiments on the formal causal inference benchmark CLadder and the logical reasoning dataset PrOntoQA show that 3B-scale language models fine-tuned with CAPT can outperform both traditional SFT and larger LLMs on in-distribution (ID) and OOD tasks.
  arXiv  Detail & Related papers  (2025-06-11T06:30:28Z)
• Statistical Hypothesis Testing for Auditing Robustness in Language Models [49.1574468325115]
  We introduce distribution-based perturbation analysis, a framework that reformulates perturbation analysis as a frequentist hypothesis testing problem.<n>We construct empirical null and alternative output distributions within a low-dimensional semantic similarity space via Monte Carlo sampling.<n>We show how we can quantify response changes, measure true/false positive rates, and evaluate alignment with reference models.
  arXiv  Detail & Related papers  (2025-06-09T17:11:07Z)
• LLM Generated Distribution-Based Prediction of US Electoral Results, Part I [0.0]
  This paper introduces distribution-based prediction, a novel approach to using Large Language Models (LLMs) as predictive tools. We demonstrate the use of distribution-based prediction in the context of recent United States presidential election.
  arXiv  Detail & Related papers  (2024-11-05T20:10:25Z)
• Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment [53.17596274334017]
  We evaluate the correctness of the detailed inference patterns of an LLM behind its seemingly correct outputs.<n>Experiments show that even when the language generation results appear correct, a significant portion of the inference patterns used by the LLM for the legal judgment may represent misleading or irrelevant logic.
  arXiv  Detail & Related papers  (2024-10-06T08:33:39Z)
• What Are the Odds? Language Models Are Capable of Probabilistic Reasoning [23.487484744911995]
  We focus on evaluating the probabilistic reasoning capabilities of language models (LMs) using idealized and real-world statistical distributions. We perform a systematic evaluation of state-of-the-art LMs on three tasks: estimating percentiles, drawing samples, and calculating probabilities.
  arXiv  Detail & Related papers  (2024-06-18T17:51:24Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration [39.35476224845088]
  Large language models (LLMs) exhibit complementary strengths in various tasks, motivating the research of LLM ensembling. We propose a training-free ensemble framework DeePEn, fusing the informative probability distributions yielded by different LLMs at each decoding step.
  arXiv  Detail & Related papers  (2024-04-19T08:52:22Z)
• Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation [73.58618024960968]
  An increasing number of studies are employing large language models (LLMs) as agents to emulate the sequential decision-making processes of humans.<n>This arouses curiosity regarding the capacity of LLM agents to comprehend probability distributions.<n>Our analysis indicates that LLM agents can understand probabilities, but they struggle with probability sampling.
  arXiv  Detail & Related papers  (2024-04-13T16:59:28Z)
• Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models [54.21695754082441]
  We propose a framework to teach Large Language Models (LLMs) to generate explainable stock predictions. A reflective agent learns how to explain past stock movements through self-reasoning, while the PPO trainer trains the model to generate the most likely explanations. Our framework can outperform both traditional deep-learning and LLM methods in prediction accuracy and Matthews correlation coefficient.
  arXiv  Detail & Related papers  (2024-02-06T03:18:58Z)
• Conformal Language Modeling [61.94417935386489]
  We propose a novel approach to conformal prediction for generative language models (LMs) Standard conformal prediction produces prediction sets with rigorous, statistical guarantees. We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation.
  arXiv  Detail & Related papers  (2023-06-16T21:55:08Z)
• ThinkSum: Probabilistic reasoning over sets using large language models [18.123895485602244]
  We propose a two-stage probabilistic inference paradigm, ThinkSum, which reasons over sets of objects or facts in a structured manner. We demonstrate the possibilities and advantages of ThinkSum on the BIG-bench suite of LLM evaluation tasks.
  arXiv  Detail & Related papers  (2022-10-04T00:34:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.