Sequential statistical inference for Large Language Models: Representation, validity, and monitoring

• URL: http://arxiv.org/abs/2606.07624v1
• Date: Sat, 30 May 2026 14:20:35 GMT
• Title: Sequential statistical inference for Large Language Models: Representation, validity, and monitoring
• Abstract summary: This discussion argues that sequential statistical inference can naturally contribute to LLM trustworthiness.<n>The discussion is organized around three tasks: modeling interactions as dependent processes; validity, developing uncertainty guarantees; and monitoring, using sequential alarms and change-point detection to identify shifts in calibration, hallucination rates, refusal behavior, fairness, or other task-relevant properties.
• Score: 10.902456009993898
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This discussion argues that sequential statistical inference can naturally contribute to LLM trustworthiness. In deployment, LLM systems are queried repeatedly, conditioned on evolving contexts, and incorporate user or tool feedback, and may exhibit behavioral shifts after model updates or distribution changes. The discussion is organized around three tasks: representation, modeling LLM interactions as dependent stochastic processes rather than isolated prompt--response pairs; validity, developing uncertainty guarantees that remain meaningful under dependence, repeated use, and adaptation; and monitoring, using sequential alarms and change-point detection to identify shifts in calibration, hallucination rates, refusal behavior, fairness, or other task-relevant properties. This perspective complements recent surveys by viewing trustworthy LLM deployment as a problem of statistical process control.

Related papers

• From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents [75.69180947909148]
  We propose a trajectory-level UQ framework for large language model (LLM) agents.<n>RuPA represents an execution history as a directed trajectory graph in which reasoning states, tool interactions, and environment feedback are nodes connected by temporal and semantic dependency edges.<n>We evaluate RUPA on representative agent benchmarks, including $$-2, Terminal-Bench-2, and GAIA, using 6 open-source LLMs spanning multiple model families.
  arXiv  Detail & Related papers  (2026-08-17T01:40:14Z)
• When Stability Fails: Hidden Failure Modes Of LLMS in Data-Constrained Scientific Decision-Making [0.0]
  Large language models (LLMs) are increasingly used as decision-support tools in data-constrained scientific work- flows.<n>We introduce a controlled behavioral evaluation framework that explicitly seprates four dimensions of LLM decision-making.<n>We show that LLMs can exhibit near-perfect run-to-run stability while diverging from statistical ground truth.
  arXiv  Detail & Related papers  (2026-03-16T19:17:09Z)
• D-Models and E-Models: Diversity-Stability Trade-offs in the Sampling Behavior of Large Language Models [91.21455683212224]
  In large language models (LLMs), the probability of relevance for the next piece of information is linked to the probability of relevance for the next product.<n>But whether fine-grained sampling probabilities faithfully align with task requirements remains an open question.<n>We identify two model types: D-models, whose P_token exhibits large step-to-step variability and poor alignment with P_task; and E-models, whose P_token is more stable and better aligned with P_task.
  arXiv  Detail & Related papers  (2026-01-25T14:59:09Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective [24.54292750583169]
  Large Language Models (LLMs) often generate responses with inherent biases, undermining their reliability in real-world applications.<n>We propose FiSCo (Fine-grained Semantic Comparison), a novel statistical framework to evaluate group-level fairness in LLMs.<n>We decompose model outputs into semantically distinct claims and apply statistical hypothesis testing to compare inter- and intra-group similarities.
  arXiv  Detail & Related papers  (2025-06-23T18:31:22Z)
• When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs [4.265831047857601]
  We investigate how large language models respond to prompts that differ only in their token-level realization but preserve the same semantic intent.<n>We propose Prompt-Based Semantic Shift (PBSS), a diagnostic framework for measuring behavioral drift in LLMs under semantically equivalent prompt rewordings.
  arXiv  Detail & Related papers  (2025-06-11T18:26:32Z)
• Statistical Hypothesis Testing for Auditing Robustness in Language Models [49.1574468325115]
  We introduce distribution-based perturbation analysis, a framework that reformulates perturbation analysis as a frequentist hypothesis testing problem.<n>We construct empirical null and alternative output distributions within a low-dimensional semantic similarity space via Monte Carlo sampling.<n>We show how we can quantify response changes, measure true/false positive rates, and evaluate alignment with reference models.
  arXiv  Detail & Related papers  (2025-06-09T17:11:07Z)
• View From Above: A Framework for Evaluating Distribution Shifts in Model Behavior [0.9043709769827437]
  Large language models (LLMs) are asked to perform certain tasks. How can we be sure that their learned representations align with reality? We propose a domain-agnostic framework for systematically evaluating distribution shifts.
  arXiv  Detail & Related papers  (2024-07-01T04:07:49Z)
• MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Dataset [50.36095192314595]
  Large Language Models (LLMs) function as conscious agents with generalizable reasoning capabilities.<n>This ability remains underexplored due to the complexity of modeling infinite possible changes in an event.<n>We introduce the first-ever benchmark, MARS, comprising three tasks corresponding to each step.
  arXiv  Detail & Related papers  (2024-06-04T08:35:04Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.