Conformalized Large Language Models under Configuration Shift

• URL: http://arxiv.org/abs/2608.01460v1
• Date: Sun, 02 Aug 2026 19:34:41 GMT
• Title: Conformalized Large Language Models under Configuration Shift
• Abstract summary: Conformal prediction is a distribution-free framework for uncertainty quantification.<n>Nonconformity scores are often induced by an inference pipeline, not just a fixed model.<n>We show that configuration shift consistently erodes CP validity.
• Score: 46.502741608809174
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Conformal prediction (CP) is a distribution-free framework for uncertainty quantification that has recently been adapted to large language models (LLMs), providing prediction sets with finite-sample coverage guarantees under exchangeability. Yet for LLMs, nonconformity scores are often induced by an inference pipeline, not just a fixed model, making them depend not only on the data distribution but also on configurable factors such as the prompt template, decoding parameters, and deployment setting. Since such configurations are routinely modified in practice but rarely treated as a source of shift, their impact on CP validity remains poorly understood. We call this \emph{configuration shift} and study it systematically along three axes: prompt template, decoding temperature, and weight quantization. In a broad empirical study spanning $9$ LLMs, $4$ datasets, and $4$ nonconformity scores, we find that configuration shift consistently erodes CP validity, often driving empirical coverage below the target. By contrast, efficiency is largely preserved: valid prediction sets remain close in size to the i.i.d. baseline. We derive coverage lower bounds that attribute this loss to a discrepancy between calibration and test score distributions, and use their finite-sample plug-in versions as empirical diagnostics of shift severity. We further show that these findings lead to practical mitigations: bound-inspired recalibration is effective with limited test examples, while fragility-aware calibration ensembling recovers much of the lost coverage without test data.

Related papers

• Robust Bayes-Assisted Conformal Prediction [0.9558392439655014]
  Bayes-assisted conformal prediction combines the strengths of Bayesian modelling with exact, distribution-free frequentist coverage guarantees.<n>We introduce RoBAS (Robust Bayes-Assisted Shrinkage): a Bayes-assisted framework for constructing robust nonconformity scores.
  arXiv  Detail & Related papers  (2026-07-05T11:26:37Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Adversarial Stress Tests for Quantum Certification [0.0]
  Apparent violations of classical benchmarks need not signal genuinely non-classical behaviour.<n>We formalise a protocol-agnostic alignment principle based on a martingale-safe lower confidence bound.<n>We show that adaptive learning-based classical agents do not enlarge the admissible classical set.
  arXiv  Detail & Related papers  (2026-03-13T03:49:58Z)
• Softmax is not Enough (for Adaptive Conformal Classification) [5.184894950445513]
  Conformal Prediction (CP) is a distribution-free framework for uncertainty quantification.<n>Nonconformity scores are derived from softmax outputs, which can be unreliable indicators of how certain the model truly is about a given input.<n>We propose a new approach that leverages information from the pre-softmax logit space, using the Helmholtz Free Energy as a measure of model uncertainty and sample difficulty.
  arXiv  Detail & Related papers  (2026-02-23T04:33:04Z)
• LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs [61.06744611795341]
  Medical vision-language models (VLMs) are strong zero-shot recognizers for medical imaging.<n>We propose texttttextbfLATA (Laplacian-Assisted Transductive Adaptation), a textittraining- and label-free refinement.<n>texttttextbfLATA sharpens zero-shot predictions without compromising exchangeability.
  arXiv  Detail & Related papers  (2026-02-19T16:45:38Z)
• BayesTTA: Continual-Temporal Test-Time Adaptation for Vision-Language Models via Gaussian Discriminant Analysis [41.09181390655176]
  Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but degrade significantly under textittemporally evolving distribution shifts common in real-world scenarios.<n>We formalize this practical problem as textitContinual-Temporal Test-Time Adaptation (CT-TTA), where test distributions evolve gradually over time.<n>We propose textitBayesTTA, a Bayesian adaptation framework that enforces temporally consistent predictions and dynamically aligns visual representations.
  arXiv  Detail & Related papers  (2025-07-11T14:02:54Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• DOTA: Distributional Test-Time Adaptation of Vision-Language Models [69.41389326333771]
  Vision-language foundation models can be unreliable when significant distribution gaps exist between training and test data.<n>We propose DOTA (DistributiOnal Test-time Adaptation), a simple yet effective method addressing this limitation.<n>This distribution-centric approach enables the model to continually learn and adapt to the deployment environment.
  arXiv  Detail & Related papers  (2024-09-28T15:03:28Z)
• Adapting Prediction Sets to Distribution Shifts Without Labels [16.478151550456804]
  We focus on a standard set-valued prediction framework called conformal prediction (CP)<n>This paper studies how to improve its practical performance using only unlabeled data from the shifted test domain.<n>We show that our methods provide consistent improvement over existing baselines and nearly match the performance of fully supervised methods.
  arXiv  Detail & Related papers  (2024-06-03T15:16:02Z)
• Robust Conformal Prediction under Distribution Shift via Physics-Informed Structural Causal Model [24.58531056536442]
  Conformal prediction (CP) handles uncertainty by predicting a set on a test input. This coverage can be guaranteed on test data even if the marginal distributions $P_X$ differ between calibration and test datasets. We propose a physics-informed structural causal model (PI-SCM) to reduce the upper bound.
  arXiv  Detail & Related papers  (2024-03-22T08:13:33Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.