Credal Large Language Models for Semantic Commitment under Uncertainty

• URL: http://arxiv.org/abs/2608.23244v1
• Date: Mon, 24 Aug 2026 13:30:55 GMT
• Title: Credal Large Language Models for Semantic Commitment under Uncertainty
• Abstract summary: Large language models (LLMs) often produce fluent but incorrect answers with unwarranted confidence.<n>Credal Large Language Models (CLLMs) induce a credal set whose lower and upper probabilities expose the spread of plausible predictive distributions.<n>We evaluate hallucination detection, calibration, selective prediction, and reasoning on Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B across OpenBookQA, CoQA, TriviaQA, and ARC-Challenge.
• Score: 7.18839452757154
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) often produce fluent but incorrect answers with unwarranted confidence. A central limitation is that standard LLMs represent uncertainty through a single predictive distribution, conflating epistemic ignorance with genuine ambiguity. We introduce Credal Large Language Models (CLLMs): an ensemble of LoRA adapters induces a credal set whose lower and upper probabilities expose the spread of plausible predictive distributions rather than collapsing to a single softmax output. From this representation we derive two complementary commitment scores. Credal Token Commitment (CTC) is a token-space score that combines lower-bound support, credal width, and intersection entropy, computed without additional generation. Semantic Commitment Consistency (SCC) extends commitment to semantic space using sampled completions, with SCC-Gap measuring the mismatch between token-level and semantic-level support. We evaluate hallucination detection, calibration, selective prediction, and reasoning on Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B across OpenBookQA, CoQA, TriviaQA, and ARC-Challenge. CLLM is the best method on QA accuracy at competitive expected calibration error, and CTC tracks the best hallucination AUROC within 1.5 pp on most settings without additional generation. On selective prediction at 80% coverage, CLLM with SCC reaches 99.0% accuracy on OpenBookQA, and on ARC-Challenge CLLM with Csem confidence achieves <= 0.6% ECE across the three backbones.

Related papers

• Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference [6.094431019524038]
  Large language model (LLM) cascades reduce inference cost by routing easy queries to a small model and deferring hard queries to a larger one.<n>We introduce textbfConformal Cascade (CC), a multi-tier inference framework that uses conformal prediction set size as the deferral rule.
  arXiv  Detail & Related papers  (2026-07-27T19:19:58Z)
• When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation [0.11280931253550518]
  Large language models (LLMs) deployed for structured generation (NER, extraction, QA, and classification) lack formal reliability guarantees.<n>We characterize when conformal risk control (CRC) can certify structured LLM outputs and when it provably cannot.
  arXiv  Detail & Related papers  (2026-06-27T19:25:07Z)
• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• Conditional Factuality Controlled LLMs with Generalization Certificates via Conformal Sampling [7.974834731988445]
  We propose emphConditional Factuality Control (CFC) for large language models (LLMs)<n>CFC defines a continuous, feature-conditional acceptance threshold through augmented quantile regression on a latent success'' score.<n>We show that CFC satisfies a conditional coverage guarantee under exchangeability and analyze its emphefficiency<n>We derive a PAC-style variant, CFC-PAC, which shrinks the nominal risk level based on a stability bound.
  arXiv  Detail & Related papers  (2026-03-28T20:48:13Z)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs [72.82403830490084]
  We argue that the decoding rule should be calibrated by correctness, not confidence alone.<n>We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps.<n>Together, our findings challenge prevailings about decoding under uncertainty and show gains across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T14:46:12Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Can Large Language Models Express Uncertainty Like Human? [71.27418419522884]
  We release the first diverse, large-scale dataset of hedging expressions with human-annotated confidence scores.<n>We conduct the first systematic study of linguistic confidence across modern large language models.
  arXiv  Detail & Related papers  (2025-09-29T02:34:30Z)
• Beyond the Score: Uncertainty-Calibrated LLMs for Automated Essay Assessment [11.525382140783043]
  This work combines conformal prediction and UAcc for essay scoring.<n> Reliability is assessed with UAcc, an uncertainty-aware accuracy that rewards models for being both correct and concise.<n>Open-source, mid-sized LLMs can already support teacher-in-the-loop AES.
  arXiv  Detail & Related papers  (2025-09-19T12:28:50Z)
• N2C2: Nearest Neighbor Enhanced Confidence Calibration for Cross-Lingual In-Context Learning [49.42251584116942]
  We conduct a thorough analysis of in-context learning (ICL) for cross-lingual scenarios.<n>ICL performs poorly in cross-lingual scenarios, exhibiting low accuracy and presenting high calibration errors.<n>We propose a novel approach, N2C2, which employs a -nearest neighbors augmented for prediction confidence calibration.
  arXiv  Detail & Related papers  (2025-03-12T10:05:05Z)
• Robust Conformal Prediction with a Single Binary Certificate [58.450154976190795]
  Conformal prediction (CP) converts any model's output to prediction sets with a guarantee to cover the true label with (adjustable) high probability.<n>We propose a robust conformal prediction that produces smaller sets even with significantly lower MC samples.
  arXiv  Detail & Related papers  (2025-03-07T08:41:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.