Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations

• URL: http://arxiv.org/abs/2204.09268v1
• Date: Wed, 20 Apr 2022 07:24:20 GMT
• Title: Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations
• Authors: Leila Pishdad, Ran Zhang, Konstantinos G. Derpanis, Allan Jepson, Afsaneh Fazly
• Abstract summary: Probabilistic embeddings have proven useful for capturing polysemous word meanings, as well as ambiguity in image matching. We propose a simple approach that replaces the standard vector point embeddings in extant image-text matching models with probabilistic distributions that are parametrically learned.
• Score: 18.560958487332265
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Probabilistic embeddings have proven useful for capturing polysemous word meanings, as well as ambiguity in image matching. In this paper, we study the advantages of probabilistic embeddings in a cross-modal setting (i.e., text and images), and propose a simple approach that replaces the standard vector point embeddings in extant image-text matching models with probabilistic distributions that are parametrically learned. Our guiding hypothesis is that the uncertainty encoded in the probabilistic embeddings captures the cross-modal ambiguity in the input instances, and that it is through capturing this uncertainty that the probabilistic models can perform better at downstream tasks, such as image-to-text or text-to-image retrieval. Through extensive experiments on standard and new benchmarks, we show a consistent advantage for probabilistic representations in cross-modal retrieval, and validate the ability of our embeddings to capture uncertainty.

Related papers

• Domain Generalization with Small Data [27.040070085669086]
  We learn a domain-invariant representation based on the probabilistic framework by mapping each data point into probabilistic embeddings. Our proposed method can marriage the measurement on the textitdistribution over distributions (i.e., the global perspective alignment) and the distribution-based contrastive semantic alignment.
  arXiv  Detail & Related papers  (2024-02-09T02:59:08Z)
• User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems [49.75149094527068]
  We show that diffusion models can be adapted to make predictions and provide uncertainty quantification for chaotic dynamical systems. We develop a probabilistic approximation scheme for the conditional score function which converges to the true distribution as the noise level decreases. We are able to sample conditionally on nonlinear userdefined events at inference time, and matches data statistics even when sampling from the tails of the distribution.
  arXiv  Detail & Related papers  (2023-06-13T03:42:03Z)
• Boosting Semi-Supervised Semantic Segmentation with Probabilistic Representations [30.672426195148496]
  We propose a Probabilistic Representation Contrastive Learning framework to improve representation quality. We define pixel-wise representations from a new perspective of probability theory. We also propose to regularize the distribution variance to enhance the reliability of representations.
  arXiv  Detail & Related papers  (2022-10-26T12:47:29Z)
• Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences [118.6018141306409]
  We propose Probabilistic Warp Consistency, a weakly-supervised learning objective for semantic matching. We first construct an image triplet by applying a known warp to one of the images in a pair depicting different instances of the same object class. Our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations.
  arXiv  Detail & Related papers  (2022-03-08T18:55:11Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• PDC-Net+: Enhanced Probabilistic Dense Correspondence Network [161.76275845530964]
  Enhanced Probabilistic Dense Correspondence Network, PDC-Net+, capable of estimating accurate dense correspondences. We develop an architecture and an enhanced training strategy tailored for robust and generalizable uncertainty prediction. Our approach obtains state-of-the-art results on multiple challenging geometric matching and optical flow datasets.
  arXiv  Detail & Related papers  (2021-09-28T17:56:41Z)
• Handling Epistemic and Aleatory Uncertainties in Probabilistic Circuits [18.740781076082044]
  We propose an approach to overcome the independence assumption behind most of the approaches dealing with a large class of probabilistic reasoning. We provide an algorithm for Bayesian learning from sparse, albeit complete, observations. Each leaf of such circuits is labelled with a beta-distributed random variable that provides us with an elegant framework for representing uncertain probabilities.
  arXiv  Detail & Related papers  (2021-02-22T10:03:15Z)
• Probabilistic Embeddings for Cross-Modal Retrieval [38.04859099157609]
  Cross-modal retrieval methods build a common representation space for samples from multiple modalities. In this paper, we argue that deterministic functions are not sufficiently powerful to capture such one-to-many correspondences. Instead, we propose to use Probabilistic Cross-Modal Embedding (PCME), where samples from the different modalities are represented as probabilistic distributions in the common embedding space.
  arXiv  Detail & Related papers  (2021-01-13T13:58:00Z)
• Learning Accurate Dense Correspondences and When to Trust Them [161.76275845530964]
  We aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. Our approach obtains state-of-the-art results on challenging geometric matching and optical flow datasets.
  arXiv  Detail & Related papers  (2021-01-05T18:54:11Z)
• Uncertainty-Aware Few-Shot Image Classification [118.72423376789062]
  Few-shot image classification learns to recognize new categories from limited labelled data. We propose Uncertainty-Aware Few-Shot framework for image classification.
  arXiv  Detail & Related papers  (2020-10-09T12:26:27Z)
• Calibrated Adversarial Refinement for Stochastic Semantic Segmentation [5.849736173068868]
  We present a strategy for learning a calibrated predictive distribution over semantic maps, where the probability associated with each prediction reflects its ground truth correctness likelihood. We demonstrate the versatility and robustness of the approach by achieving state-of-the-art results on the multigrader LIDC dataset and on a modified Cityscapes dataset with injected ambiguities. We show that the core design can be adapted to other tasks requiring learning a calibrated predictive distribution by experimenting on a toy regression dataset.
  arXiv  Detail & Related papers  (2020-06-23T16:39:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.