Localized Conformal Prediction for Image Classification with Vision-Language Models

• URL: http://arxiv.org/abs/2606.31577v1
• Date: Tue, 30 Jun 2026 12:34:40 GMT
• Title: Localized Conformal Prediction for Image Classification with Vision-Language Models
• Abstract summary: We show that simple usage of the cosine similarity between test-time and calibration visual features is not sufficient to improve over the non-local baselines.<n>We propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction.
• Score: 5.212510010425201
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees. Full conditional guarantee is not an attainable goal, a well known fact in conformal predictions literature. As a result, several approaches have tried to approximate this behavior by adapting the conformal sets of test-time samples according to their similarity to calibration examples. Although the latter has gained traction and shown impressive performances for regression problems, its application to image classification remains under-explored. We conduct an extensive benchmarking on natural image classification tasks with vision-language models (VLMs), using our open source implementation of a recent localized conformal prediction algorithm. We show that straightforward usage of the cosine similarity between test-time and calibration visual features, an intuitive choice for VLMs, is not sufficient to improve over the non-local baselines. In response, we propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction. Code is available at https://github.com/cfuchs2023/lcp-vlm/.

Related papers

• Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration [36.91587844807066]
  Multimodal Large Language Models (MLLMs) have shown strong performance in multi-image cross-modal retrieval.<n>Yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance.<n>We propose a training-free, attention-guided debiasing framework that leverages intrinsic attention signals for instance-level correction.
  arXiv  Detail & Related papers  (2026-05-12T06:17:01Z)
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models [86.8650252164764]
  Large language models (LLMs) are prone to generating factually incorrect outputs.<n>We propose an adaptive conformal prediction approach that extends conformal score transformation methods to LLMs.<n>This enables prompt-dependent calibration, retaining marginal coverage guarantees while improving conditional coverage.
  arXiv  Detail & Related papers  (2026-04-15T15:35:42Z)
• DANCE: Doubly Adaptive Neighborhood Conformal Estimation [12.643121779828526]
  We propose a doubly locally adaptive nearest-neighbor based conformal algorithm combining two novel nonconformity scores directly using the data's embedded representation.<n>We test against state-of-the-art local, task-adapted and zero-shot conformal baselines, demonstrating DANCE's superior blend of set size efficiency and robustness across various datasets.
  arXiv  Detail & Related papers  (2026-02-24T07:54:53Z)
• Conformal Prediction Sets for Instance Segmentation [10.042082668896038]
  We introduce a conformal prediction algorithm to generate adaptive confidence sets for instance segmentation.<n>Given an image and a pixel coordinate query, our algorithm generates a confidence set of instance predictions for that pixel.<n>We apply our algorithm to instance segmentation examples in agricultural field delineation, cell segmentation, and vehicle detection.
  arXiv  Detail & Related papers  (2026-02-10T18:15:06Z)
• Sparse Activations as Conformal Predictors [19.298282860984116]
  We find a novel connection between conformal prediction and sparse softmax-like transformations.<n>We introduce new non-conformity scores for classification that make the calibration process correspond to the widely used temperature scaling method.<n>We show that the proposed method achieves competitive results in terms of coverage, efficiency, and adaptiveness.
  arXiv  Detail & Related papers  (2025-02-20T17:53:41Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples.<n>We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Post-hoc Probabilistic Vision-Language Models [54.05237186168399]
  Vision-language models (VLMs) have found remarkable success in classification, retrieval, and generative tasks.<n>We propose post-hoc uncertainty estimation in VLMs that does not require additional training.<n>Our results show promise for safety-critical applications of large-scale models.
  arXiv  Detail & Related papers  (2024-12-08T18:16:13Z)
• Contrastive Model Adaptation for Cross-Condition Robustness in Semantic Segmentation [58.17907376475596]
  We investigate normal-to-adverse condition model adaptation for semantic segmentation. Our method -- CMA -- leverages such image pairs to learn condition-invariant features via contrastive learning. We achieve state-of-the-art semantic segmentation performance for model adaptation on several normal-to-adverse adaptation benchmarks.
  arXiv  Detail & Related papers  (2023-03-09T11:48:29Z)
• Uncertainty Sets for Image Classifiers using Conformal Prediction [112.54626392838163]
  We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%. The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset. Our method modifies an existing conformal prediction algorithm to give more stable predictive sets by regularizing the small scores of unlikely classes after Platt scaling.
  arXiv  Detail & Related papers  (2020-09-29T17:58:04Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.