Language Model Probabilities are Not Calibrated in Numeric Contexts

• URL: http://arxiv.org/abs/2410.16007v2
• Date: Tue, 04 Mar 2025 19:14:05 GMT
• Title: Language Model Probabilities are Not Calibrated in Numeric Contexts
• Authors: Charles Lovering, Michael Krumdick, Viet Dac Lai, Seth Ebner, Nilesh Kumar, Varshini Reddy, Rik Koncel-Kedziorski, Chris Tanner,
• Abstract summary: We argue that language model (LM) outputs should capture natural distributions.<n>Our work specifically tests whether LM output probabilities are calibrated to numeric information within their textual contexts.
• Score: 16.17638166383352
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Some statements have one well-defined continuation (e.g., "the Eiffel Tower is in [Paris]"), whereas others have a natural distribution over multiple options (e.g., "the weighted coin flip was [Heads/Tails].") We argue that language model (LM) outputs should capture these natural distributions. Our work specifically tests whether LM output probabilities are calibrated to numeric information within their textual contexts. For example, if the context (the prompt) concerns two equally likely options (e.g., heads or tails for a fair coin), the LM output probabilities should also be equal. Likewise, in a context with nonuniformly likely events (e.g., rolling a pair with two dice) an LM should output proportionate probabilities. However, we find that even in simple settings, the best LMs (1) are poorly calibrated and (2) have systematic biases: artifacts like word identity, word order, and word frequency all impact calibration. For example, gpt-4o-mini often picks the first of two options presented in the prompt regardless of the options' implied likelihoods, whereas Llama-3.1-8B picks the second. Models do not allocate probability mass among valid options in a calibrated manner.

Related papers

• Calibrated Large Language Models for Binary Question Answering [49.1574468325115]
  A well-calibrated model should produce probabilities that accurately reflect the likelihood of its predictions being correct. We propose a novel approach that utilizes the inductive Venn--Abers predictor (IVAP) to calibrate the probabilities associated with the output tokens corresponding to the binary labels.
  arXiv  Detail & Related papers  (2024-07-01T09:31:03Z)
• What Are the Odds? Language Models Are Capable of Probabilistic Reasoning [23.487484744911995]
  We focus on evaluating the probabilistic reasoning capabilities of language models (LMs) using idealized and real-world statistical distributions. We perform a systematic evaluation of state-of-the-art LMs on three tasks: estimating percentiles, drawing samples, and calculating probabilities.
  arXiv  Detail & Related papers  (2024-06-18T17:51:24Z)
• "My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models [40.867655189493924]
  Open-ended nature of language generation makes evaluation of large language models (LLMs) challenging. One common evaluation approach uses multiple-choice questions (MCQ) to limit the response space. We evaluate how aligned first-token evaluation is with the text output along several dimensions.
  arXiv  Detail & Related papers  (2024-02-22T12:47:33Z)
• Language Models with Conformal Factuality Guarantees [44.767328168194815]
  Conformal factuality is a framework that can ensure high probability correctness guarantees for language model (LM) outputs. We show that conformal prediction in language models corresponds to a back-off algorithm that provides high probability correctness guarantees.
  arXiv  Detail & Related papers  (2024-02-15T18:31:53Z)
• Large Language Models Are Not Robust Multiple Choice Selectors [117.72712117510953]
  Multiple choice questions (MCQs) serve as a common yet important task format in the evaluation of large language models (LLMs) This work shows that modern LLMs are vulnerable to option position changes due to their inherent "selection bias" We propose a label-free, inference-time debiasing method, called PriDe, which separates the model's prior bias for option IDs from the overall prediction distribution.
  arXiv  Detail & Related papers  (2023-09-07T17:44:56Z)
• Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts [119.22672589020394]
  We propose COnfidence-baSed MOdel Selection (CosMoS), where model confidence can effectively guide model selection. We evaluate CosMoS on four datasets with spurious correlations, each with multiple test sets with varying levels of data distribution shift.
  arXiv  Detail & Related papers  (2023-06-19T18:48:15Z)
• Conformal Language Modeling [61.94417935386489]
  We propose a novel approach to conformal prediction for generative language models (LMs) Standard conformal prediction produces prediction sets with rigorous, statistical guarantees. We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation.
  arXiv  Detail & Related papers  (2023-06-16T21:55:08Z)
• Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling [95.9542389945259]
  Sentence scoring aims at measuring the likelihood of a sentence and is widely used in many natural language processing scenarios. We propose textitTranscormer -- a Transformer model with a novel textitsliding language modeling (SLM) for sentence scoring.
  arXiv  Detail & Related papers  (2022-05-25T18:00:09Z)
• Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases [55.45617404586874]
  We propose a few-shot instruction-based method for prompting pre-trained language models (LMs) We show that large LMs can detect different types of fine-grained biases with similar and sometimes superior accuracy to fine-tuned models.
  arXiv  Detail & Related papers  (2021-12-15T04:19:52Z)
• Thought Flow Nets: From Single Predictions to Trains of Model Thought [39.619001911390804]
  When humans solve complex problems, they rarely come up with a decision right-away. Instead, they start with an intuitive decision reflecting upon it, spot mistakes, resolve contradictions and jump between different hypotheses.
  arXiv  Detail & Related papers  (2021-07-26T13:56:37Z)
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering [80.82194311274694]
  We examine the question "how can we know when language models know, with confidence, the answer to a particular query?" We examine three strong generative models -- T5, BART, and GPT-2 -- and study whether their probabilities on QA tasks are well calibrated. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness.
  arXiv  Detail & Related papers  (2020-12-02T03:53:13Z)
• A Note on High-Probability versus In-Expectation Guarantees of Generalization Bounds in Machine Learning [95.48744259567837]
  Statistical machine learning theory often tries to give generalization guarantees of machine learning models. Statements made about the performance of machine learning models have to take the sampling process into account. We show how one may transform one statement to another.
  arXiv  Detail & Related papers  (2020-10-06T09:41:35Z)
• Learning Posterior and Prior for Uncertainty Modeling in Person Re-Identification [11.651410633259543]
  We learn the sample posterior and the class prior distribution in the latent space, so that not only representative features but also the uncertainty can be built by the model. Experiments have been carried out on Market1501, DukeMTMC, MARS and noisy dataset as well.
  arXiv  Detail & Related papers  (2020-07-17T07:20:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.