Large Language Models Develop Belief State Geometry In-Context

• URL: http://arxiv.org/abs/2609.17376v1
• Date: Tue, 15 Sep 2026 16:17:59 GMT
• Title: Large Language Models Develop Belief State Geometry In-Context
• Abstract summary: Large language models (LLMs) trained on next-token prediction exhibit remarkable in-context learning abilities.<n>We consider such representations in a controlled setting, prompting LLMs with data emitted from hidden Markov models.<n>We find that belief states are linearly decodable from residual stream activations.
• Score: 1.6972273159174482
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) trained on next-token prediction exhibit remarkable in-context learning (ICL) abilities, yet the representations that support ICL remain poorly understood. We consider such representations in a controlled setting: prompting LLMs with data emitted from hidden Markov models (HMMs) and probing for the corresponding belief state -- the posterior distribution over the HMM's hidden states given the observed token history. Across six open-source LLMs prompted with data from 40 HMMs selected for non-trivial belief structure, we find that belief states are linearly decodable from residual stream activations, with peak probe $R^2$-values from 0.83-0.99 across HMM and LLM combinations, ranging from early to late layers. To establish functional relevance, we intervene directly on the probe-identified subspace via patching and steering, resulting in downstream prediction quality on the order of the untampered model, while controls degrade performance substantially. Together, these results provide representation-level evidence that ICL in open-source LLMs approximates optimal Bayesian prediction over a context-inferred generative model. More broadly, our findings extend prior results linking input-distribution structure to activation geometry: from toy networks trained explicitly on HMM data to production-scale LLMs.

Related papers

• Extracting Algorithms in Pre-trained LLMs: A Case on Hidden Markov Models [12.294879009851476]
  Large language models (LLMs) display a striking ability to predict next observations from Hidden Markov Models (HMMs) via in-context learning (ICL)<n>We close this gap with a three-stage pipeline. First, we empirically compare LLM behavior against a suite of candidate algorithms and narrow the space to three classes.<n>Second, we derive theoretical connections between the three classes and show how each can be implemented in-context by a Transformer.<n>Third, returning to pre-trained LLMs, we introduce the Principal Activations Probe (PAP), a layer-wise probing and intervention method that isolates algorithmic signals in model activation
  arXiv  Detail & Related papers  (2026-06-24T17:42:19Z)
• What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization [5.663538370244175]
  We propose LoID, a deterministic method for extracting informative prior distributions for Bayesian logistic regression.<n>Rather than relying on generated text, we probe the model's confidence in opposing semantic directions through carefully constructed sentences.<n>We evaluate LoID on ten real-world datasets under synthetic out-of-distribution (OOD) settings.
  arXiv  Detail & Related papers  (2026-01-24T22:05:01Z)
• Large Language Models as Universal Predictors? An Empirical Study on Small Tabular Datasets [0.0]
  Large Language Models (LLMs) can perform predictive tasks over structured inputs without explicit fine-tuning on downstream tasks.<n>We investigate the empirical function approximation capability of LLMs on small-scale structured datasets for classification, regression and clustering tasks.<n>Our findings suggest that LLMs can serve as general-purpose predictive engines for structured data, with clear strengths in classification and significant limitations in regression and clustering.
  arXiv  Detail & Related papers  (2025-08-24T15:00:51Z)
• Beyond Naïve Prompting: Strategies for Improved Zero-shot Context-aided Forecasting with LLMs [57.82819770709032]
  Large language models (LLMs) can be effective context-aided forecasters via na"ive direct prompting.<n>ReDP improves interpretability by eliciting explicit reasoning traces, allowing us to assess the model's reasoning over the context.<n>CorDP leverages LLMs solely to refine existing forecasts with context, enhancing their applicability in real-world forecasting pipelines.<n> IC-DP proposes embedding historical examples of context-aided forecasting tasks in the prompt, substantially improving accuracy even for the largest models.
  arXiv  Detail & Related papers  (2025-08-13T16:02:55Z)
• Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision [50.45597801390757]
  Instruct-LF is a goal-oriented latent factor discovery system.<n>It integrates instruction-following ability with statistical models to handle noisy datasets.
  arXiv  Detail & Related papers  (2025-02-21T02:03:08Z)
• LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization [59.75242204923353]
  We introduce LLM-Lasso, a framework that leverages large language models (LLMs) to guide feature selection in Lasso regression.<n>LLMs generate penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model.<n>Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model.
  arXiv  Detail & Related papers  (2025-02-15T02:55:22Z)
• Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models [49.48313161005423]
  A hybrid language model (HLM) architecture integrates a small language model (SLM) operating on a mobile device with a large language model (LLM) hosted at the base station (BS) of a wireless network.<n>The HLM token generation process follows the speculative inference principle: the SLM's vocabulary distribution is uploaded to the LLM, which either accepts or rejects it, with rejected tokens being resampled by the LLM.<n>We propose a novel HLM structure coined Uncertainty-aware opportunistic HLM (U-HLM), wherein the SLM locally measures its output uncertainty and skips both up
  arXiv  Detail & Related papers  (2024-12-17T09:08:18Z)
• Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output [49.893971654861424]
  We present a light-weight approach for detecting nonfactual outputs from retrieval-augmented generation (RAG) We compute a factuality score that can be thresholded to yield a binary decision. Our experiments show high area under the ROC curve (AUC) across a wide range of relevant open source datasets.
  arXiv  Detail & Related papers  (2024-11-01T20:44:59Z)
• A Comprehensive Evaluation of Large Language Models on Temporal Event Forecasting [45.0261082985087]
  We conduct a comprehensive evaluation of Large Language Models (LLMs) for temporal event forecasting.<n>We find that fine-tuning LLMs with raw texts can significantly improve performance.<n>However, issues such as popularity bias and the long-tail problem persist in LLMs.
  arXiv  Detail & Related papers  (2024-07-16T11:58:54Z)
• PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks [13.503004451037212]
  This paper introduces PhyloLM, a method adapting phylogenetic algorithms to Large Language Models (LLMs)<n>Our method calculates a phylogenetic distance metric based on the similarity of LLMs' output.<n>Our phylogenetic distance predicts performance in standard benchmarks, thus demonstrating its functional validity.
  arXiv  Detail & Related papers  (2024-04-06T16:16:30Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.