The Reliability Paradox: Exploring How Shortcut Learning Undermines Language Model Calibration

• URL: http://arxiv.org/abs/2412.15269v1
• Date: Tue, 17 Dec 2024 08:04:28 GMT
• Title: The Reliability Paradox: Exploring How Shortcut Learning Undermines Language Model Calibration
• Authors: Geetanjali Bihani, Julia Rayz,
• Abstract summary: Pre-trained language models (PLMs) have enabled significant performance gains in the field of natural language processing. Recent studies have found PLMs to suffer from miscalibration, indicating a lack of accuracy in the confidence estimates provided by these models. This paper investigates whether lower calibration error implies reliable decision rules for a language model.
• Score: 5.616884466478886
• License:
• Abstract: The advent of pre-trained language models (PLMs) has enabled significant performance gains in the field of natural language processing. However, recent studies have found PLMs to suffer from miscalibration, indicating a lack of accuracy in the confidence estimates provided by these models. Current evaluation methods for PLM calibration often assume that lower calibration error estimates indicate more reliable predictions. However, fine-tuned PLMs often resort to shortcuts, leading to overconfident predictions that create the illusion of enhanced performance but lack generalizability in their decision rules. The relationship between PLM reliability, as measured by calibration error, and shortcut learning, has not been thoroughly explored thus far. This paper aims to investigate this relationship, studying whether lower calibration error implies reliable decision rules for a language model. Our findings reveal that models with seemingly superior calibration portray higher levels of non-generalizable decision rules. This challenges the prevailing notion that well-calibrated models are inherently reliable. Our study highlights the need to bridge the current gap between language model calibration and generalization objectives, urging the development of comprehensive frameworks to achieve truly robust and reliable language models.

Related papers

• Finetuning Language Models to Emit Linguistic Expressions of Uncertainty [5.591074369497796]
  Large language models (LLMs) are increasingly employed in information-seeking and decision-making tasks. LLMs tend to generate information that conflicts with real-world facts, and their persuasive style can make these inaccuracies appear confident and convincing. In this work, we explore supervised finetuning on uncertainty-augmented predictions as a method to develop models that produce linguistic expressions of uncertainty.
  arXiv  Detail & Related papers  (2024-09-18T17:52:53Z)
• Enhancing Healthcare LLM Trust with Atypical Presentations Recalibration [20.049443396032423]
  Black-box large language models (LLMs) are increasingly deployed in various environments. LLMs often exhibit overconfidence, leading to potential risks and misjudgments. We propose a novel method, textitAtypical presentations Recalibration, which leverages atypical presentations to adjust the model's confidence estimates.
  arXiv  Detail & Related papers  (2024-09-05T03:45:35Z)
• Large Language Models Must Be Taught to Know What They Don't Know [97.90008709512921]
  We show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead. We also investigate the mechanisms that enable reliable uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators.
  arXiv  Detail & Related papers  (2024-06-12T16:41:31Z)
• Calibrating Large Language Models with Sample Consistency [76.23956851098598]
  We explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. Results show that consistency-based calibration methods outperform existing post-hoc approaches. We offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.
  arXiv  Detail & Related papers  (2024-02-21T16:15:20Z)
• Selective Learning: Towards Robust Calibration with Dynamic Regularization [79.92633587914659]
  Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. We introduce Dynamic Regularization (DReg) which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off.
  arXiv  Detail & Related papers  (2024-02-13T11:25:20Z)
• A Study on the Calibration of In-context Learning [27.533223818505682]
  We study in-context learning (ICL), a prevalent method for adapting static language models through tailored prompts. We observe that, with an increasing number of ICL examples, models initially exhibit increased miscalibration before achieving better calibration. We explore recalibration techniques and find that a scaling-binning calibrator can reduce calibration errors consistently.
  arXiv  Detail & Related papers  (2023-12-07T03:37:39Z)
• On the Calibration of Large Language Models and Alignment [63.605099174744865]
  Confidence calibration serves as a crucial tool for gauging the reliability of deep models. We conduct a systematic examination of the calibration of aligned language models throughout the entire construction process. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.
  arXiv  Detail & Related papers  (2023-11-22T08:57:55Z)
• Towards Calibrated Robust Fine-Tuning of Vision-Language Models [97.19901765814431]
  This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. We show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value.
  arXiv  Detail & Related papers  (2023-11-03T05:41:25Z)
• Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Models [23.881825575095945]
  Large pre-trained language models (PLMs) have demonstrated strong performance on natural language understanding (NLU) tasks through fine-tuning. However, fine-tuned models still suffer from overconfident predictions, especially in out-of-domain settings. We demonstrate that the PLMs are well-calibrated on the masked language modeling task with robust predictive confidence under domain shift. We show that preserving pre-trained features can improve the calibration of fine-tuned language models.
  arXiv  Detail & Related papers  (2023-05-30T17:35:31Z)
• On the Calibration of Massively Multilingual Language Models [15.373725507698591]
  Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer. We first investigate the calibration of MMLMs in the zero-shot setting and observe a clear case of miscalibration in low-resource languages. We also find that few-shot examples in the language can further help reduce the calibration errors, often substantially.
  arXiv  Detail & Related papers  (2022-10-21T21:41:56Z)
• On the Inference Calibration of Neural Machine Translation [54.48932804996506]
  We study the correlation between calibration and translation performance and linguistic properties of miscalibration. We propose a new graduated label smoothing method that can improve both inference calibration and translation performance.
  arXiv  Detail & Related papers  (2020-05-03T02:03:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.