Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective

• URL: http://arxiv.org/abs/2406.00793v1
• Date: Sun, 2 Jun 2024 16:20:30 GMT
• Title: Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective
• Abstract summary: In-context learning (ICL) has emerged as a remarkable characteristic of Large Language Models (LLM) We show that the martingale property is a necessary condition for unambiguous predictions in such scenarios. We derive actionable checks with corresponding theory and test statistics which must hold if the martingale property is satisfied.
• Score: 4.415371871652809
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context learning (ICL) has emerged as a particularly remarkable characteristic of Large Language Models (LLM): given a pretrained LLM and an observed dataset, LLMs can make predictions for new data points from the same distribution without fine-tuning. Numerous works have postulated ICL as approximately Bayesian inference, rendering this a natural hypothesis. In this work, we analyse this hypothesis from a new angle through the martingale property, a fundamental requirement of a Bayesian learning system for exchangeable data. We show that the martingale property is a necessary condition for unambiguous predictions in such scenarios, and enables a principled, decomposed notion of uncertainty vital in trustworthy, safety-critical systems. We derive actionable checks with corresponding theory and test statistics which must hold if the martingale property is satisfied. We also examine if uncertainty in LLMs decreases as expected in Bayesian learning when more data is observed. In three experiments, we provide evidence for violations of the martingale property, and deviations from a Bayesian scaling behaviour of uncertainty, falsifying the hypothesis that ICL is Bayesian.

Related papers

• BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation [20.375742002723765]
  Large language models (LLMs) are typically deployed in multi-turn conversations.<n>We ask how closely LLMs' belief updates match those of a rational Bayesian reasoner in multi-turn settings.<n>We introduce BayesBench, a suite of simulation environments that probe this across three progressively complex tasks.
  arXiv  Detail & Related papers  (2026-06-29T19:30:32Z)
• Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning [26.42437186150084]
  We propose a systematic evaluation framework for belief entrenchment in large language models.<n>We use the Martingale property from Bayesian statistics to measure violations of this property.<n>We identify the models, reasoning techniques, and domains more prone to belief entrenchment.
  arXiv  Detail & Related papers  (2025-12-02T16:34:05Z)
• The Consistency Hypothesis in Uncertainty Quantification for Large Language Models [22.60039074743706]
  Black-box uncertainty quantification (UQ) methods, relying solely on model API access, have gained popularity due to their practical benefits.<n>In this paper, we examine the implicit assumption behind several UQ methods, which use generation consistency as a proxy for confidence.<n>We propose data-free black-box UQ methods that aggregate similarities between generations for confidence estimation.
  arXiv  Detail & Related papers  (2025-06-27T01:53:15Z)
• Pretrained LLMs Learn Multiple Types of Uncertainty [23.807232455808613]
  Large Language Models are known to capture real-world knowledge, allowing them to excel in many downstream tasks.<n>In this work, we study how well LLMs capture uncertainty, without explicitly being trained for that.<n>We show that, if considering uncertainty as a linear concept in the model's latent space, it might indeed be captured, even after only pretraining.
  arXiv  Detail & Related papers  (2025-05-27T14:06:15Z)
• Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models [79.76293901420146]
  Large Language Models (LLMs) are employed across various high-stakes domains, where the reliability of their outputs is crucial. Our research investigates the fragility of uncertainty estimation and explores potential attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which, when activated by a specific trigger in the input, manipulates the model's uncertainty without affecting the final output.
  arXiv  Detail & Related papers  (2024-07-15T23:41:11Z)
• Quantifying Prediction Consistency Under Model Multiplicity in Tabular LLMs [10.494477811252034]
  Fine-tuning large language models can lead to textitfine-tuning multiplicity, where equally well-performing models make conflicting predictions on the same inputs. This raises critical concerns about the robustness and reliability of Tabular LLMs. This work proposes a novel metric to quantify the robustness of individual predictions without expensive model retraining.
  arXiv  Detail & Related papers  (2024-07-04T22:22:09Z)
• Calibrated Large Language Models for Binary Question Answering [49.1574468325115]
  A well-calibrated model should produce probabilities that accurately reflect the likelihood of its predictions being correct. We propose a novel approach that utilizes the inductive Venn--Abers predictor (IVAP) to calibrate the probabilities associated with the output tokens corresponding to the binary labels.
  arXiv  Detail & Related papers  (2024-07-01T09:31:03Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Uncertainty Quantification for In-Context Learning of Large Language Models [52.891205009620364]
  In-context learning has emerged as a groundbreaking ability of Large Language Models (LLMs) We propose a novel formulation and corresponding estimation method to quantify both types of uncertainties. The proposed method offers an unsupervised way to understand the prediction of in-context learning in a plug-and-play fashion.
  arXiv  Detail & Related papers  (2024-02-15T18:46:24Z)
• A Hypothesis-Driven Framework for the Analysis of Self-Rationalising Models [0.8702432681310401]
  We use a Bayesian network to implement a hypothesis about how a task is solved. The resulting models do not exhibit a strong similarity to GPT-3.5. We discuss the implications of this as well as the framework's potential to approximate LLM decisions better in future work.
  arXiv  Detail & Related papers  (2024-02-07T12:26:12Z)
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [69.83976050879318]
  In large language models (LLMs), identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions.
  arXiv  Detail & Related papers  (2023-11-15T05:58:35Z)
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning [123.3472310767721]
  Prediction credibility measures are fundamental in statistics and machine learning. These measures should account for the wide variety of models used in practice. The framework developed in this work expresses the credibility as a risk-fit trade-off.
  arXiv  Detail & Related papers  (2020-11-24T19:52:38Z)
• Getting a CLUE: A Method for Explaining Uncertainty Estimates [30.367995696223726]
  We propose a novel method for interpreting uncertainty estimates from differentiable probabilistic models. Our method, Counterfactual Latent Uncertainty Explanations (CLUE), indicates how to change an input, while keeping it on the data manifold.
  arXiv  Detail & Related papers  (2020-06-11T21:53:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.