Post-hoc Probabilistic Vision-Language Models

• URL: http://arxiv.org/abs/2412.06014v1
• Date: Sun, 08 Dec 2024 18:16:13 GMT
• Title: Post-hoc Probabilistic Vision-Language Models
• Authors: Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Arno Solin, Martin Trapp,
• Abstract summary: Vision-language models (VLMs) have found remarkable success in classification, retrieval, and generative tasks. We propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our results show promise for safety-critical applications of large-scale models.
• Score: 51.12284891724463
• License:
• Abstract: Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs deterministically map images and text descriptions to a joint latent space in which their similarity is assessed using the cosine similarity. However, a deterministic mapping of inputs fails to capture uncertainties over concepts arising from domain shifts when used in downstream tasks. In this work, we propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our method leverages a Bayesian posterior approximation over the last layers in VLMs and analytically quantifies uncertainties over cosine similarities. We demonstrate its effectiveness for uncertainty quantification and support set selection in active learning. Compared to baselines, we obtain improved and well-calibrated predictive uncertainties, interpretable uncertainty estimates, and sample-efficient active learning. Our results show promise for safety-critical applications of large-scale models.

Related papers

• BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptation [30.435971066422706]
  We show that the discriminative performance of state-of-the-art CLIP adapters does not always correlate with their uncertainty estimation capabilities. We introduce BayesAdapter, which leverages Bayesian inference to estimate a full probability distribution instead of a single point. Our approach obtains high quality uncertainty estimates in the predictions, standing out in calibration and selective classification.
  arXiv  Detail & Related papers  (2024-12-12T20:48:06Z)
• Uncertainty-boosted Robust Video Activity Anticipation [72.14155465769201]
  Video activity anticipation aims to predict what will happen in the future, embracing a broad application prospect ranging from robot vision to autonomous driving. Despite the recent progress, the data uncertainty issue, reflected as the content evolution process and dynamic correlation in event labels, has been somehow ignored. We propose an uncertainty-boosted robust video activity anticipation framework, which generates uncertainty values to indicate the credibility of the anticipation results.
  arXiv  Detail & Related papers  (2024-04-29T12:31:38Z)
• Introspective Deep Metric Learning [91.47907685364036]
  We propose an introspective deep metric learning framework for uncertainty-aware comparisons of images. The proposed IDML framework improves the performance of deep metric learning through uncertainty modeling.
  arXiv  Detail & Related papers  (2023-09-11T16:21:13Z)
• Uncertainty-aware LiDAR Panoptic Segmentation [21.89063036529791]
  We introduce a novel approach for solving the task of uncertainty-aware panoptic segmentation using LiDAR point clouds. Our proposed EvLPSNet network is the first to solve this task efficiently in a sampling-free manner. We provide several strong baselines combining state-of-the-art panoptic segmentation networks with sampling-free uncertainty estimation techniques.
  arXiv  Detail & Related papers  (2022-10-10T07:54:57Z)
• BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networks [50.15201777970128]
  We propose BayesCap that learns a Bayesian identity mapping for the frozen model, allowing uncertainty estimation. BayesCap is a memory-efficient method that can be trained on a small fraction of the original dataset. We show the efficacy of our method on a wide variety of tasks with a diverse set of architectures.
  arXiv  Detail & Related papers  (2022-07-14T12:50:09Z)
• Which models are innately best at uncertainty estimation? [15.929238800072195]
  Deep neural networks must be equipped with an uncertainty estimation mechanism when deployed for risk-sensitive tasks. This paper studies the relationship between deep architectures and their training regimes with their corresponding selective prediction and uncertainty estimation performance.
  arXiv  Detail & Related papers  (2022-06-05T11:15:35Z)
• Learning Accurate Dense Correspondences and When to Trust Them [161.76275845530964]
  We aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. Our approach obtains state-of-the-art results on challenging geometric matching and optical flow datasets.
  arXiv  Detail & Related papers  (2021-01-05T18:54:11Z)
• Uncertainty-Aware Few-Shot Image Classification [118.72423376789062]
  Few-shot image classification learns to recognize new categories from limited labelled data. We propose Uncertainty-Aware Few-Shot framework for image classification.
  arXiv  Detail & Related papers  (2020-10-09T12:26:27Z)
• Probabilistic Deep Learning for Instance Segmentation [9.62543698736491]
  We propose a generic method to obtain model-inherent uncertainty estimates within proposal-free instance segmentation models. We evaluate our method on the BBBC010 C. elegans dataset, where it yields competitive performance.
  arXiv  Detail & Related papers  (2020-08-24T19:51:48Z)
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation [27.077741143188867]
  We propose a family of algorithms which split the classification task into two stages: representation learning and uncertainty estimation. We evaluate their performance in terms of emphselective classification (risk-coverage), and their ability to detect out-of-distribution samples.
  arXiv  Detail & Related papers  (2020-01-22T15:08:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.