Task-Aware Calibration: Provably Optimal Decoding in LLMs

• URL: http://arxiv.org/abs/2605.10202v1
• Date: Mon, 11 May 2026 08:48:04 GMT
• Title: Task-Aware Calibration: Provably Optimal Decoding in LLMs
• Abstract summary: LLM decoding often relies on the model's predictive distribution to generate an output.<n>We introduce task calibration as a paradigm to calibrate the model's predictive distribution in the task-induced latent space.<n>Our work demonstrates that task calibration enables more reliable model decisions across various tasks and applications.
• Score: 46.26328678647223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM decoding often relies on the model's predictive distribution to generate an output. Consequently, misalignment with respect to the true generating distribution leads to suboptimal decisions in practice. While a natural solution is to calibrate the model's output distribution, for LLMs, this is ill-posed at the combinatorially vast level of free-form language. We address this by building on the insight that in many tasks, these free-form outputs can be interpreted in a semantically meaningful latent structure, for example, discrete class labels, integers, or sets. We introduce task calibration as a paradigm to calibrate the model's predictive distribution in the task-induced latent space. We apply a decision-theoretic result to show that Minimum Bayes Risk (MBR) decoding on the task-calibrated latent distribution is the optimal decoding strategy on latent model beliefs. Empirically, it consistently improves generation quality across different tasks and baselines. We also introduce Task Calibration Error (TCE), an application-aware calibration metric that quantifies the excess loss due to miscalibration. Our work demonstrates that task calibration enables more reliable model decisions across various tasks and applications.

Related papers

• Multi-Bitwidth Quantization for LLMs Using Additive Codebooks [12.237109162791091]
  Large language models (LLMs) are increasingly deployed across heterogeneous hardware with varying resource constraints.<n>We propose Drop-by-Drop, a novel multi-bitwidth post-training quantization framework that enables inference-time precision control over LLM weights from a single trained model.
  arXiv  Detail & Related papers  (2026-06-11T04:06:02Z)
• Model Merging by Output-Space Projection [6.752805486904705]
  Model merging combines fine-tuned checkpoints into a single multi-task model without retraining.<n>We show that merging can be formulated as a convex programme over residual updates.<n>We extend to multi-layer merging via a sequential layer-wise algorithm and demonstrate consistent gains across language and vision benchmarks.
  arXiv  Detail & Related papers  (2026-05-27T21:01:29Z)
• Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision [16.289117637700446]
  A canonical mitigation method is to provide calibrated confidence scores that faithfully reflect their likelihood of correctness at the instance-level.<n>Our study proposes local Platt-scaling applied separately to three different fine-grained confidence scores.<n>We find that fine-grained confidence scores consistently achieve lower calibration error across a broader range of probability intervals.
  arXiv  Detail & Related papers  (2026-04-08T06:41:09Z)
• Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume [45.38219855706969]
  We introduce UMPIRE, a training-free uncertainty quantification framework for Multimodal Large Language Models (MLLMs)<n>UMPIRE computes the incoherence-adjusted semantic volume of sampled MLLM responses for a given task instance.<n>We show that UMPIRE consistently outperforms baseline metrics in error detection and uncertainty calibration across image, audio, and video-text benchmarks.
  arXiv  Detail & Related papers  (2026-02-27T17:18:42Z)
• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Learning What Matters: Probabilistic Task Selection via Mutual Information for Model Finetuning [20.93518809718398]
  We introduce TASKPGM, a principled and scalable framework for mixture optimization.<n> TASKPGM selects continuous task proportions by minimizing an energy function over a Markov Random Field (MRF)<n>Our method yields a closed form solution under simplex constraints and provably balances representativeness and diversity among tasks.
  arXiv  Detail & Related papers  (2025-07-16T20:14:55Z)
• Calibrating Uncertainty Quantification of Multi-Modal LLMs using Grounding [48.92310906093414]
  We introduce a novel approach for calibrating uncertainty quantification (UQ) tailored for multi-modal large language models (LLMs)<n>We leverage cross-modal consistency in addition to self-consistency to improve the calibration of the multi-modal models.<n>We evaluate the proposed approach across multiple multi-modal tasks, such as medical question answering (Slake) and visual question answering (VQAv2), considering multi-modal models such as LLaVA-Med and LLaVA.
  arXiv  Detail & Related papers  (2025-04-30T19:19:21Z)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment [90.6117569025754]
  Reinforcement learning from human feedback has emerged as an effective technique to align Large Language models.<n>Controlled Decoding provides a mechanism for aligning a model at inference time without retraining.<n>We propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies.
  arXiv  Detail & Related papers  (2025-03-27T17:34:25Z)
• Adaptive Draft-Verification for Efficient Large Language Model Decoding [24.347886232342862]
  Large language model (LLM) decoding involves generating a sequence of tokens based on a given context. The typical autoregressive decoding method requires a separate forward pass through the model for each token generated. We introduce ADED, which accelerates LLM decoding without requiring fine-tuning.
  arXiv  Detail & Related papers  (2024-06-27T22:20:39Z)
• Calibrating Large Language Models with Sample Consistency [76.23956851098598]
  We explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. Results show that consistency-based calibration methods outperform existing post-hoc approaches. We offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.
  arXiv  Detail & Related papers  (2024-02-21T16:15:20Z)
• Large Language Models are Miscalibrated In-Context Learners [22.30783674111999]
  In this work, we deliver an in-depth analysis of the behavior across different choices of learning methods.<n>We observe that the miscalibration problem exists across all learning methods in low-resource setups.<n>We find that self-ensembling with max probability produces robust and calibrated predictions.
  arXiv  Detail & Related papers  (2023-12-21T11:55:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.