A CAP-like Trilemma for Large Language Models: Correctness, Non-bias, and Utility under Semantic Underdetermination

• URL: http://arxiv.org/abs/2605.11672v1
• Date: Tue, 12 May 2026 07:28:38 GMT
• Title: A CAP-like Trilemma for Large Language Models: Correctness, Non-bias, and Utility under Semantic Underdetermination
• Abstract summary: This paper formulates a CAP-like conjecture for Large Language Models (LLMs)<n>The proposed trilemma states that, under semantic underdetermination, an LLM cannot always simultaneously guarantee strong correctness, strict non-bias, and high utility.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The CAP theorem states that a distributed system cannot simultaneously guarantee consistency, availability, and partition tolerance under network partition. Inspired by this result, this paper formulates a CAP-like conjecture for Large Language Models (LLMs). The proposed trilemma states that, under semantic underdetermination, an LLM cannot always simultaneously guarantee strong correctness, strict non-bias, and high utility. A prompt is semantically underdetermined when the given premises do not determine a unique answer. In such cases, a useful and decisive response requires the model to introduce a selection criterion, preference, prior, or value ordering. If this criterion is not supplied by the user or justified by the available premises, the response becomes biased in a broad selection-theoretic sense. Conversely, if the model avoids unsupported preferences, it may preserve correctness and non-bias but may reduce utility through refusal, hedging, or clarification. The paper formalizes this correctness--non-bias--utility trilemma, develops examples, and argues that certain LLM failures arise not merely from model limitations but from the structure of underdetermined decision requests.

Related papers

• Preference Reasoning under Indeterminacy in Large Language Models [36.06390570712731]
  We argue that indeterminacy, rather than correctness alone, is a central challenge for AI reasoning.<n>We show that state-of-the-art language models systematically fail to distinguish between determined and undetermined instances.
  arXiv  Detail & Related papers  (2026-08-19T07:27:12Z)
• SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment [20.5795766462733]
  We propose a group-level uncertainty target that constructs an answer-conditioned uncertainty geometry over sampled responses.<n>Experiments show improved uncertainty ranking, lower calibration error, and reduced overconfidence.
  arXiv  Detail & Related papers  (2026-06-09T23:17:16Z)
• Clarify, Abstain or Answer? Strategising in Conversation with Belief-Augmented Generation [38.81368744366676]
  Large language models (LLMs) define a distribution over text, which can be viewed as a probabilistic representation of uncertainty.<n>We propose Belief-Augmented Generation (BAG): grounding LLMs in their own belief state via the prompt and letting them reason over these K samples to decide on a conversational strategy.<n>BAG improves QA accuracy across six models and yields strategy decisions more faithful to the belief state than prompt-only baselines.
  arXiv  Detail & Related papers  (2026-05-25T13:29:02Z)
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models [86.8650252164764]
  Large language models (LLMs) are prone to generating factually incorrect outputs.<n>We propose an adaptive conformal prediction approach that extends conformal score transformation methods to LLMs.<n>This enables prompt-dependent calibration, retaining marginal coverage guarantees while improving conditional coverage.
  arXiv  Detail & Related papers  (2026-04-15T15:35:42Z)
• Semantic Self-Distillation for Language Model Uncertainty [19.97226069762587]
  We show that lightweight student models can estimate a prompt-conditioned uncertainty before a language model generates an answer token.<n>The entropy of this distribution provides an effective uncertainty signal for hallucination prediction and the probability density allows candidate answers to be evaluated for reliability.<n>On TriviaQA, our student models match or outperform finite-sample semantic dispersion for hallucination prediction and provide a strong signal for out-of-domain answer detection.
  arXiv  Detail & Related papers  (2026-02-04T14:03:28Z)
• Non-Resolution Reasoning (NRR): A Computational Framework for Contextual Identity and Ambiguity Preservation [0.0]
  Current artificial intelligence systems exhibit a fundamental architectural limitation: they resolve ambiguity prematurely.<n>This premature semantic collapse stems from classical identity assumptions embedded in standard neural architectures.<n>We propose Non-Resolution Reasoning (NRR), a computational framework that treats ambiguity retention as a valid reasoning mode.
  arXiv  Detail & Related papers  (2025-12-15T16:14:32Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• The Illusion of Certainty: Uncertainty quantification for LLMs fails under ambiguity [48.899855816199484]
  We introduce MAQA* and AmbigQA*, the first ambiguous question-answering (QA) datasets equipped with ground-truth answer distributions.<n>We show that predictive-distribution and ensemble-based estimators are fundamentally limited under ambiguity.
  arXiv  Detail & Related papers  (2025-11-06T14:46:35Z)
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling [46.23327887393273]
  We introduce a diversity-steered sampler that discourages semantically redundant outputs during decoding.<n>Key idea is to inject a continuous semantic-similarity penalty into the model's proposal distribution.<n>Being modular and requiring no gradient access to the base LLM, the framework promises to serve as a drop-in enhancement for uncertainty estimation.
  arXiv  Detail & Related papers  (2025-10-24T10:06:21Z)
• SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models [0.27309692684728604]
  Large Language Models (LLMs) can achieve inflated scores on multiple-choice tasks by exploiting inherent biases in option positions or labels.<n>This study introduces SCOPE, an evaluation framework designed to measure and mitigate such selection bias in a dataset-independent manner.
  arXiv  Detail & Related papers  (2025-07-24T08:28:17Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.