Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees

• URL: http://arxiv.org/abs/2508.05544v1
• Date: Thu, 07 Aug 2025 16:22:49 GMT
• Title: Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees
• Abstract summary: We propose a frequency-based uncertainty quantification method under black-box settings.<n>Our approach involves multiple independent samplings of the model's output distribution for each input.<n>We show that frequency-based PE outperforms logit-based PE in distinguishing between correct and incorrect predictions.
• Score: 5.09580026885155
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have shown remarkable progress in multiple-choice question answering (MCQA), but their inherent unreliability, such as hallucination and overconfidence, limits their application in high-risk domains. To address this, we propose a frequency-based uncertainty quantification method under black-box settings, leveraging conformal prediction (CP) to ensure provable coverage guarantees. Our approach involves multiple independent samplings of the model's output distribution for each input, with the most frequent sample serving as a reference to calculate predictive entropy (PE). Experimental evaluations across six LLMs and four datasets (MedMCQA, MedQA, MMLU, MMLU-Pro) demonstrate that frequency-based PE outperforms logit-based PE in distinguishing between correct and incorrect predictions, as measured by AUROC. Furthermore, the method effectively controls the empirical miscoverage rate under user-specified risk levels, validating that sampling frequency can serve as a viable substitute for logit-based probabilities in black-box scenarios. This work provides a distribution-free model-agnostic framework for reliable uncertainty quantification in MCQA with guaranteed coverage, enhancing the trustworthiness of LLMs in practical applications.

Related papers

• LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy [1.3254304182988286]
  We propose Adaptive Conformal Semantic Entropy (ACSE), a method for estimating prompt-level uncertainty.<n>Our uncertainty scoring function is based on clustering semantic entropy of multiple diverse responses to the same prompt.<n>Our approach consistently outperforms state-of-the-art uncertainty quantification baselines.
  arXiv  Detail & Related papers  (2026-05-05T20:56:11Z)
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models [86.8650252164764]
  Large language models (LLMs) are prone to generating factually incorrect outputs.<n>We propose an adaptive conformal prediction approach that extends conformal score transformation methods to LLMs.<n>This enables prompt-dependent calibration, retaining marginal coverage guarantees while improving conditional coverage.
  arXiv  Detail & Related papers  (2026-04-15T15:35:42Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMs [20.08099668437471]
  multimodal large language models (MLLMs) generated predictions can enable selective prediction and improve user confidence.<n>We propose Functionally Equivalent Sampling for Trust Assessment (FESTA), a multimodal input sampling technique for MLLMs.<n>FESTA generates an uncertainty measure based on the equivalent and complementary input samplings.
  arXiv  Detail & Related papers  (2025-09-20T11:50:22Z)
• Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control [0.0]
  This study introduces a significance testing-enhanced conformal prediction (CP) framework to improve trustworthiness of large language models (LLMs) in multiple-choice question answering (MCQA)<n>CP provides statistically rigorous marginal coverage guarantees for prediction sets, and significance testing offers established statistical rigor.<n>This work establishes a statistical framework for trustworthy LLM deployment in high-stakes QA applications.
  arXiv  Detail & Related papers  (2025-08-07T16:46:47Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction [0.0]
  We propose a model-agnostic uncertainty quantification method that integrates dynamic threshold calibration and cross-modal consistency verification.<n>We show that the framework achieves stable performance across varying calibration-to-test split ratios, underscoring its robustness for real-world deployment in healthcare, autonomous systems, and other safety-sensitive domains.<n>This work bridges the gap between theoretical reliability and practical applicability in multi-modal AI systems, offering a scalable solution for hallucination detection and uncertainty-aware decision-making.
  arXiv  Detail & Related papers  (2025-04-24T15:39:46Z)
• Correctness Coverage Evaluation for Medical Multiple-Choice Question Answering Based on the Enhanced Conformal Prediction Framework [2.9599960287815144]
  Large language models (LLMs) are increasingly adopted in medical question-answering (QA) scenarios.<n>LLMs can generate hallucinations and nonfactual information, undermining their trustworthiness in high-stakes medical tasks.<n>This paper proposes an enhanced Conformal Prediction framework for medical multiple-choice question-answering tasks.
  arXiv  Detail & Related papers  (2025-03-07T15:22:10Z)
• Rectifying Conformity Scores for Better Conditional Coverage [75.73184036344908]
  We present a new method for generating confidence sets within the split conformal prediction framework.<n>Our method performs a trainable transformation of any given conformity score to improve conditional coverage while ensuring exact marginal coverage.
  arXiv  Detail & Related papers  (2025-02-22T19:54:14Z)
• Online scalable Gaussian processes with conformal prediction for guaranteed coverage [32.21093722162573]
  The consistency of the resulting uncertainty values hinges on the premise that the learning function conforms to the properties specified by the GP model. We propose to wed the GP with the prevailing conformal prediction (CP), a distribution-free post-processing framework that produces it prediction sets with a provably valid coverage.
  arXiv  Detail & Related papers  (2024-10-07T19:22:15Z)
• Probabilistic Conformal Prediction with Approximate Conditional Validity [81.30551968980143]
  We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution. Our method consistently outperforms existing approaches in terms of conditional coverage.
  arXiv  Detail & Related papers  (2024-07-01T20:44:48Z)
• ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees [68.33498595506941]
  We introduce a novel uncertainty measure based on self-consistency theory. We then develop a conformal uncertainty criterion by integrating the uncertainty condition aligned with correctness into the CP algorithm. Empirical evaluations indicate that our uncertainty measure outperforms prior state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-06-29T17:33:07Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• Conformal Prediction for Federated Uncertainty Quantification Under Label Shift [57.54977668978613]
  Federated Learning (FL) is a machine learning framework where many clients collaboratively train models. We develop a new conformal prediction method based on quantile regression and take into account privacy constraints.
  arXiv  Detail & Related papers  (2023-06-08T11:54:58Z)
• A Semi-Bayesian Nonparametric Estimator of the Maximum Mean Discrepancy Measure: Applications in Goodness-of-Fit Testing and Generative Adversarial Networks [3.623570119514559]
  We propose a semi-Bayesian nonparametric (semi-BNP) procedure for the goodness-of-fit (GOF) test. Our method introduces a novel Bayesian estimator for the maximum mean discrepancy (MMD) measure. We demonstrate that our proposed test outperforms frequentist MMD-based methods by achieving a lower false rejection and acceptance rate of the null hypothesis.
  arXiv  Detail & Related papers  (2023-03-05T10:36:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.