Closing the Calibration Gap in Semantic Caching

• URL: http://arxiv.org/abs/2606.19719v2
• Date: Fri, 19 Jun 2026 01:10:41 GMT
• Title: Closing the Calibration Gap in Semantic Caching
• Abstract summary: We show that semantic caching models with the highest PR-AUC are often the worst in operation.<n>We introduce Precision-Cache Hit Ratio (P-CHR) AUC, a cache-aware metric that measures precision across cache utilization levels, and Retention Rate (CRR)<n>Our experiments show that the calibration gap is governed by the training objective rather than data scale, and post-hoc calibration only partially closes it.
• Score: 0.07456526005219319
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these systems using PR-AUC, a metric that only measures how well scores rank and ignores whether they are usable at a fixed threshold. We show this mismatch leads to systematically poor deployment choices, as models with the highest PR-AUC are often the worst in operation. We introduce Precision-Cache Hit Ratio (P-CHR) AUC, a cache-aware metric that measures precision across cache utilization levels, and Calibration Retention Rate (CRR), which captures how much offline ranking quality survives at deployment. We decompose the operational gap between offline and deployed quality into a recoverable calibration component and an irreducible structural component fixed by the dataset's positive rate. Our experiments show that the calibration gap is governed by the training objective rather than data scale, and post-hoc calibration only partially closes it. Ultimately, model selection for semantic caching is a calibration problem, not a ranking one, and measuring it is the first step to closing the gap.

Related papers

• Retrieval-Corrected Conformal Prediction for Time Series [31.867011243141818]
  Conformal prediction (CP) provides distribution-free prediction intervals for fixed forecasters.<n>Recent time series CP methods improve local calibration using recent, weighted, or localized residuals.<n>Local calibration can remain indirect, since broad residual weighting or additional adaptation procedures may dilute the evidence most relevant to the current prediction.<n>We propose Retrieval-Corrected Conformal Prediction (RCCP), a retrieval-augmented calibration method for time series prediction intervals.
  arXiv  Detail & Related papers  (2026-08-11T06:43:01Z)
• Rethinking Post-Hoc Calibration in Semantic Segmentation [2.8002121919083685]
  Post-hoc calibration offers a practical way to correct confidence estimates without retraining the segmentation model.<n>We study two such issues: first, adding a constant to all logits leaves the softmax probabilities unchanged, but several standard calibrators can still depend on this arbitrary offset.<n>Second, post-hoc calibration is typically fitted by minimizing a likelihood-based objective, whereas segmentation models are trained with task-specific metrics such as Dice.
  arXiv  Detail & Related papers  (2026-07-02T09:01:21Z)
• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors [22.39558434131574]
  We introduce the concept of consistency as an alternative perspective on model calibration. We propose a post-hoc calibration method called Consistency (CC) which adjusts confidence based on the model's consistency across inputs. We show that performing perturbations at the logit level significantly improves computational efficiency.
  arXiv  Detail & Related papers  (2024-10-16T06:55:02Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Threshold-Consistent Margin Loss for Open-World Deep Metric Learning [42.03620337000911]
  Existing losses used in deep metric learning (DML) for image retrieval often lead to highly non-uniform intra-class and inter-class representation structures. Inconsistency often complicates the threshold selection process when deploying commercial image retrieval systems. We propose a novel variance-based metric called Operating-Point-Inconsistency-Score (OPIS) that quantifies the variance in the operating characteristics across classes.
  arXiv  Detail & Related papers  (2023-07-08T21:16:41Z)
• Least Square Calibration for Peer Review [18.063450032460047]
  We propose a flexible framework, namely least square calibration (LSC), for selecting top candidates from peer ratings. Our framework provably performs perfect calibration from noiseless linear scoring functions under mild assumptions. Our algorithm consistently outperforms the baseline which select top papers based on the highest average ratings.
  arXiv  Detail & Related papers  (2021-10-25T02:40:33Z)
• Localized Calibration: Metrics and Recalibration [133.07044916594361]
  We propose a fine-grained calibration metric that spans the gap between fully global and fully individualized calibration. We then introduce a localized recalibration method, LoRe, that improves the LCE better than existing recalibration methods.
  arXiv  Detail & Related papers  (2021-02-22T07:22:12Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.