Auditing Question-Order Effects in Large Language Models with the QQ Equality: Mechanism Characterization and a Saturation Caveat

• URL: http://arxiv.org/abs/2607.17219v1
• Date: Sun, 19 Jul 2026 12:22:06 GMT
• Title: Auditing Question-Order Effects in Large Language Models with the QQ Equality: Mechanism Characterization and a Saturation Caveat
• Abstract summary: Human survey respondents exhibit question-order effects that satisfy the QQ (quantum question) equality.<n>We develop the QQ equality into an audit criterion for sequential judgments of autoregressive large language models.
• Score: 4.07636450847048
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human survey respondents exhibit question-order effects that satisfy the QQ (quantum question) equality, an a priori, parameter-free prediction of the projective quantum question-order model. We develop the QQ equality into an audit criterion for sequential judgments of autoregressive large language models (LLMs). Theoretically, we characterize which mechanism classes satisfy it robustly: marginal-independent kernels satisfy QQ iff all four mismatch transition rates coincide (a class containing the 2D rank-1 projective model with a fixed measurement pair under state variation); a polarity- and position-dependent repetition family is characterized by an exact cross-symmetry condition with closed-form violations; QQ-satisfying behaviors are closed under order-matched mixing; and the rank-2 Contextuality-by-Default criterion translates into audit coordinates as $|\qQQ|\le\OSS$, where $\OSS$ (the order-sensitivity score) totals the order sensitivity of the two marginals. Methodologically, we develop a pre-specified, audit-logged pipeline applicable to any model exposing next-token log-probabilities; it combines worst-case robustness envelopes, sampling-consistency spot checks, full label counterbalancing, and a saturation diagnostic. Empirically, in a first-signal pilot on an open-weight instruction-tuned model under two framings, all pre-specified health gates passed, yet 17/18 and 7/8 item pairs, respectively, were saturated (near-deterministic), and no item was certified residually contextual. Forced-binary next-token log-probabilities were thus inadequate for distribution-level QQ audits under the tested model and prompting conditions; we recommend pre-specified saturation diagnostics whenever next-token distributions are treated as survey-response distributions.

Related papers

• From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs [0.9751538234290761]
  Missing fabrications can harm users and institutions, while false alarms consume limited human-review capacity.<n>We study two signals accessible through black-box model APIs: semantic entropy, and uncertainty derived from token log-probabilities.<n>We extend token-based uncertainty detection by aggregating token-level signals across sampled responses through our TopK method.
  arXiv  Detail & Related papers  (2026-09-02T14:50:24Z)
• Probabilistic Model Checking of Autoregressive Neural Sequence Models [4.570003973862485]
  Test-set accuracy is silent on two issues that matter when deploying autoregressive neural sequence models.<n>How much probability mass the system under test places on constraint-violating alternatives that are reachable under sampling.<n>We answer both with probabilistic model checking.
  arXiv  Detail & Related papers  (2026-09-01T07:37:55Z)
• Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading [0.6916773850242582]
  Order-agnostic language models (OALMs) are trained to predict masked tokens under arbitrary conditioning sets.<n>We show that learning conditionals are not exact factorizations of a coherent joint distribution.<n>We propose a complementary diagnostic based on the shape of the confidence trace.
  arXiv  Detail & Related papers  (2026-05-31T04:25:36Z)
• A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering [19.55210880950831]
  Expected measures whether a model's predicted confidence aligns with its empirical accuracy, and is central to the reliable deployment of large language models (LLMs)<n>We introduce Sem-ECE, a calibration evaluation framework for open-ended question answering (QA)<n>We study two estimators within framework: Sem$$-ECE, the same-sample self-consistency score, and Sem$$-ECE, a held-out variant that separates answer selection from confidence evaluation.
  arXiv  Detail & Related papers  (2026-05-08T19:53:49Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• From Calibration to Refinement: Seeking Certainty via Probabilistic Evidence Propagation for Noisy-Label Person Re-Identification [40.73759251488672]
  Existing noise-robust person Re-ID methods rely on loss-correction or sample-selection strategies using softmax outputs.<n>We propose the CAlibration-to-REfinement (CARE) method, a two-stage framework that seeks certainty through probabilistic evidence propagation from calibration to refinement.<n>In the refinement stage, we design the evidence propagation refinement (EPR) that can more accurately distinguish between clean and noisy samples.
  arXiv  Detail & Related papers  (2026-02-26T15:50:15Z)
• Statistical Inference for Matching Decisions via Matrix Completion under Dependent Missingness [6.28693385008859]
  We propose an algorithm for non-optimal entry for three canonical mechanisms, i.e. one-to-one, one-sided matching with one-sided random arrival, and two-sided random arrival.<n>Our empirical experiments demonstrate that the proposed approach provides accurate estimation valid confidence intervals and efficient evaluation.
  arXiv  Detail & Related papers  (2025-10-30T13:31:32Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Mitigating LLM Hallucinations via Conformal Abstention [70.83870602967625]
  We develop a principled procedure for determining when a large language model should abstain from responding in a general domain. We leverage conformal prediction techniques to develop an abstention procedure that benefits from rigorous theoretical guarantees on the hallucination rate (error rate) Experimentally, our resulting conformal abstention method reliably bounds the hallucination rate on various closed-book, open-domain generative question answering datasets.
  arXiv  Detail & Related papers  (2024-04-04T11:32:03Z)
• Regularized Vector Quantization for Tokenized Image Synthesis [126.96880843754066]
  Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while quantization suffers from low codebook utilization and reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate perturbed above issues effectively by applying regularization from two perspectives.
  arXiv  Detail & Related papers  (2023-03-11T15:20:54Z)
• Breaking the Spurious Causality of Conditional Generation via Fairness Intervention with Corrective Sampling [77.15766509677348]
  Conditional generative models often inherit spurious correlations from the training dataset. This can result in label-conditional distributions that are imbalanced with respect to another latent attribute. We propose a general two-step strategy to mitigate this issue.
  arXiv  Detail & Related papers  (2022-12-05T08:09:33Z)
• Distribution-free uncertainty quantification for classification under label shift [105.27463615756733]
  We focus on uncertainty quantification (UQ) for classification problems via two avenues. We first argue that label shift hurts UQ, by showing degradation in coverage and calibration. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
  arXiv  Detail & Related papers  (2021-03-04T20:51:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.