When are likely answers right? On Sequence Probability and Correctness in LLMs

• URL: http://arxiv.org/abs/2606.27359v1
• Date: Thu, 25 Jun 2026 17:58:02 GMT
• Title: When are likely answers right? On Sequence Probability and Correctness in LLMs
• Abstract summary: When does sequence probability, that is, the conditional probability of a continuation given a prompt, actually align with correctness?<n>We quantify this relationship across decoding methods, models, and benchmarks at four levels.<n>We find that higher sequence probability is often predictive of correctness across prompt-answer pairs within a fixed dataset.
• Score: 38.14443290666347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model, either locally at the token level or globally at the sequence level. Therefore, their success depends on a fundamental question: when does sequence probability, that is, the conditional probability of a continuation given a prompt, actually align with correctness? In this paper, we set out to quantify this relationship across decoding methods, models, and benchmarks at four levels: across decoding methods, across hyperparameters within a method, across prompt-answer pairs within a dataset, and across repeated responses to the same prompt. We find that higher sequence probability is often predictive of correctness across prompt-answer pairs within a fixed dataset. However, this relationship does not generally transfer to decoding decisions: increasing sequence probability by changing hyperparameters or methods does not reliably improve accuracy. Further, sequence probability is not a good indicator of correctness for responses to the same prompt. These findings clarify when decoding can and cannot be expected to improve correctness, and provide practical guidance for decoding, self-consistency, and verifier-free self-improvement.

Related papers

• From token probabilities to calibrated confidence: An empirical study of mathematical question answering [1.0032961794537367]
  Prior work has shown that token probabilities are often overconfident.<n>We compare single-pass estimators, which reuse token probabilities from the original generation, with multi-pass estimators.<n>We find that aggregating token probabilities over the full sequence captures small but consistent differences between correct and incorrect generations.
  arXiv  Detail & Related papers  (2026-08-08T00:00:19Z)
• Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading [0.6916773850242582]
  Order-agnostic language models (OALMs) are trained to predict masked tokens under arbitrary conditioning sets.<n>We show that learning conditionals are not exact factorizations of a coherent joint distribution.<n>We propose a complementary diagnostic based on the shape of the confidence trace.
  arXiv  Detail & Related papers  (2026-05-31T04:25:36Z)
• Latent Veracity Inference for Identifying Errors in Stepwise Reasoning [78.29317733206643]
  We introduce Veracity Search (VS), a discrete search algorithm over veracity assignments.<n>It performs otherwise intractable inference in the posterior distribution over latent veracity values.<n>It generalizes VS, enabling accurate zero-shot veracity inference in novel contexts.
  arXiv  Detail & Related papers  (2025-05-17T04:16:36Z)
• DiffSampling: Enhancing Diversity and Accuracy in Neural Text Generation [5.734448042909701]
  We propose DiffSampling, a new decoding method that leverages a mathematical analysis of the token probability distribution.<n>Experiments involving four different text-generation tasks demonstrate that our approach consistently performs at least on par with the existing methods.
  arXiv  Detail & Related papers  (2025-02-19T19:00:02Z)
• Large language model validity via enhanced conformal prediction methods [4.081098869497239]
  We develop new conformal inference methods for obtaining validity guarantees on the output of large language models (LLMs) We generalize the conditional conformal procedure of Gibbs et al. (2023) in order to adaptively issue weaker guarantees when they are required to preserve the utility of the output. We demonstrate the efficacy of our approach on biography and medical question-answering datasets.
  arXiv  Detail & Related papers  (2024-06-14T04:46:39Z)
• Predicting generalization performance with correctness discriminators [56.948321040125734]
  We present a novel model that establishes upper and lower bounds on the accuracy, without requiring gold labels for the unseen data.<n>We show across a variety of tagging, parsing, and semantic parsing tasks that the gold accuracy is reliably between the predicted upper and lower bounds.
  arXiv  Detail & Related papers  (2023-11-15T22:43:42Z)
• GRACE: Discriminator-Guided Chain-of-Thought Reasoning [75.35436025709049]
  We propose Guiding chain-of-thought ReAsoning with a CorrectnEss Discriminator (GRACE) to steer the decoding process towards producing correct reasoning steps. GRACE employs a discriminator trained with a contrastive loss over correct and incorrect steps, which is used during decoding to score next-step candidates.
  arXiv  Detail & Related papers  (2023-05-24T09:16:51Z)
• Performance-guaranteed regularization in maximum likelihood method: Gauge symmetry in Kullback -- Leibler divergence [0.0]
  Regularization is similar to error-correcting codes, which obtain optimal decoding by mixing suboptimal solutions with an incorrectly received code. We propose a theoretically guaranteed regularization in the maximum likelihood method by focusing on a gauge symmetry in Kullback -- Leibler divergence.
  arXiv  Detail & Related papers  (2023-03-29T14:17:21Z)
• Calibrating AI Models for Wireless Communications via Conformal Prediction [55.47458839587949]
  Conformal prediction is applied for the first time to the design of AI for communication systems. This paper investigates the application of conformal prediction as a general framework to obtain AI models that produce decisions with formal calibration guarantees.
  arXiv  Detail & Related papers  (2022-12-15T12:52:23Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering [80.82194311274694]
  We examine the question "how can we know when language models know, with confidence, the answer to a particular query?" We examine three strong generative models -- T5, BART, and GPT-2 -- and study whether their probabilities on QA tasks are well calibrated. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness.
  arXiv  Detail & Related papers  (2020-12-02T03:53:13Z)
• Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction [0.8594140167290097]
  We develop conformal prediction methods for constructing valid confidence sets in multiclass and multilabel problems. By leveraging ideas from quantile regression, we build methods that always guarantee correct coverage but additionally provide conditional coverage for both multiclass and multilabel prediction problems.
  arXiv  Detail & Related papers  (2020-04-21T17:45:38Z)
• Optimal Change-Point Detection with Training Sequences in the Large and Moderate Deviations Regimes [72.68201611113673]
  This paper investigates a novel offline change-point detection problem from an information-theoretic perspective. We assume that the knowledge of the underlying pre- and post-change distributions are not known and can only be learned from the training sequences which are available.
  arXiv  Detail & Related papers  (2020-03-13T23:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.