Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

• URL: http://arxiv.org/abs/2607.15277v1
• Date: Thu, 16 Jul 2026 17:59:31 GMT
• Title: Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models
• Abstract summary: Law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population.<n>We use binary trees as an evaluation scaffold to partition a population into increasingly fine-grained subpopulations.<n>We prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity.
• Score: 45.42736932931535
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In-context learning is commonly interpreted as a form of conditional inference, in which the prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Applying this protocol across problem domains and state-of-the-art frontier models, we show widespread violations of basic consistency properties. An in-depth study of persona prompting reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. This effect persists across variations in tree structure and estimation task, and can be partially recovered through implicit prompting. Together, these findings suggest that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.

Related papers

• Generalized Score Matching for Parameter Estimation on Convex Domains [13.954816561132942]
  We derive the generalized score matching objective on a convex subset of $mathbbRd$.<n>We show how classical score matching as well as domain-adapted variants for non-negative data arise naturally within the proposed framework.<n>Our derivation sheds new light on the scope and applicability of generalized score matching in various problem settings.
  arXiv  Detail & Related papers  (2026-09-10T13:23:41Z)
• Rethinking the Evaluation and Optimization of LLM-Based Social Simulation [56.18577997572273]
  We use the subjectivity coefficient to analyze how accuracy-based evaluation and hard-label training fail as subjectivity grows.<n>We construct SUBJSIM, a benchmark of 19,300 contexts covering 193 annotators and 100 subjective questions.<n>Results on SUBJSIM demonstrate the advantages of our method.
  arXiv  Detail & Related papers  (2026-08-20T06:26:03Z)
• Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases [0.0]
  We introduce an analytically exact framework for the controlled behavioral evaluation of Large Language Models (LLMs)<n>We replace unstructured prompting with fully crossed factorial experiments to systematically isolate causal main and interaction effects.<n>We derive a multivariate ordinal consensus metric and a distributional ANOVA to process these PMFs analytically.<n>We validate our framework with a case study on consumer ethnocentrism across five LLMs, demonstrating how our approach isolates systemic country-of-origin biases that aggregate benchmarks otherwise obscure.
  arXiv  Detail & Related papers  (2026-08-11T05:20:06Z)
• Beyond Averages: Evaluating LLMs on Human Survey Replication at the Distributional Level [8.995779634140495]
  We evaluate survey replication using a non-public 2010 consumer choice experiment on Korean instant noodle purchases.<n>We compare human and LLM responses at mean-level, pattern, and distributional alignment, and against reference baselines from the human data alone.
  arXiv  Detail & Related papers  (2026-06-08T04:22:15Z)
• Evidence-based Distributional Alignment for Large Language Models [58.65469623911573]
  We propose Evi-DA, an evidence-based alignment technique that improves the fidelity and robustness of LLM-based distribution estimation.<n>Given a target country, Evi-DA retrieves related World Values Survey items and their answer distributions, predicts a coarse Welzel value signature for each option, and infers the country-conditioned answer distribution in a structured format.
  arXiv  Detail & Related papers  (2026-03-03T03:34:06Z)
• Differential privacy with dependent data [1.8835490533310795]
  We show that Winsorized mean estimators can be used under dependence for bounded data.<n>We formalize dependence via log-Sobolev inequalities on the joint unbounded observations.<n>Our work constitutes a first step towards a systematic study of Differential Privacy (DP) for dependent data.
  arXiv  Detail & Related papers  (2025-11-23T18:56:40Z)
• The Illusion of Certainty: Uncertainty quantification for LLMs fails under ambiguity [48.899855816199484]
  We introduce MAQA* and AmbigQA*, the first ambiguous question-answering (QA) datasets equipped with ground-truth answer distributions.<n>We show that predictive-distribution and ensemble-based estimators are fundamentally limited under ambiguity.
  arXiv  Detail & Related papers  (2025-11-06T14:46:35Z)
• MF-LLM: Simulating Population Decision Dynamics via a Mean-Field Large Language Model Framework [53.82097200295448]
  Mean-Field LLM (MF-LLM) is first to incorporate mean field theory into social simulation.<n>MF-LLM models bidirectional interactions between individuals and the population through an iterative process.<n> IB-Tune is a novel fine-tuning method inspired by the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-04-30T12:41:51Z)
• Benchmarking Distributional Alignment of Large Language Models [43.0198231524816]
  Language models (LMs) are increasingly used as simulacra for people, yet their ability to match the distribution of views of a specific demographic group remains uncertain. We construct a dataset expanding beyond political values, create human baselines for this task, and evaluate the extent to which an LM can align with a particular group's opinion distribution. Our analysis reveals open problems regarding if, and how, LMs can be used to simulate humans, and that LLMs can more accurately describe the opinion distribution than simulate such distributions.
  arXiv  Detail & Related papers  (2024-11-08T08:41:17Z)
• Evaluating statistical language models as pragmatic reasoners [39.72348730045737]
  We evaluate the capacity of large language models to infer meanings of pragmatic utterances. We find that LLMs can derive context-grounded, human-like distributions over the interpretations of several complex pragmatic utterances. Results inform the inferential capacity of statistical language models, and their use in pragmatic and semantic parsing applications.
  arXiv  Detail & Related papers  (2023-05-01T18:22:10Z)
• Evaluating Distributional Distortion in Neural Language Modeling [81.83408583979745]
  A heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language. Standard language modeling metrics such as perplexity quantify the performance of language models (LM) in aggregate. We develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages.
  arXiv  Detail & Related papers  (2022-03-24T01:09:46Z)
• Counterfactual Maximum Likelihood Estimation for Training Deep Networks [83.44219640437657]
  Deep learning models are prone to learning spurious correlations that should not be learned as predictive clues. We propose a causality-based training framework to reduce the spurious correlations caused by observable confounders. We conduct experiments on two real-world tasks: Natural Language Inference (NLI) and Image Captioning.
  arXiv  Detail & Related papers  (2021-06-07T17:47:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.