Probabilistic and Semantic Descriptions of Image Manifolds and Their Applications

• URL: http://arxiv.org/abs/2307.02881v5
• Date: Sun, 12 Nov 2023 01:52:35 GMT
• Title: Probabilistic and Semantic Descriptions of Image Manifolds and Their Applications
• Authors: Peter Tu, Zhaoyuan Yang, Richard Hartley, Zhiwei Xu, Jing Zhang, Yiwei Fu, Dylan Campbell, Jaskirat Singh, Tianyu Wang
• Abstract summary: It is common to say that images lie on a lower-dimensional manifold in the high-dimensional space. Images are unevenly distributed on the manifold, and our task is to devise ways to model this distribution as a probability distribution. We show how semantic interpretations are used to describe points on the manifold.
• Score: 28.554065677506966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper begins with a description of methods for estimating image probability density functions that reflects the observation that such data is usually constrained to lie in restricted regions of the high-dimensional image space-not every pattern of pixels is an image. It is common to say that images lie on a lower-dimensional manifold in the high-dimensional space. However, it is not the case that all points on the manifold have an equal probability of being images. Images are unevenly distributed on the manifold, and our task is to devise ways to model this distribution as a probability distribution. We therefore consider popular generative models. For our purposes, generative/probabilistic models should have the properties of 1) sample generation: the possibility to sample from this distribution with the modelled density function, and 2) probability computation: given a previously unseen sample from the dataset of interest, one should be able to compute its probability, at least up to a normalising constant. To this end, we investigate the use of methods such as normalising flow and diffusion models. We then show how semantic interpretations are used to describe points on the manifold. To achieve this, we consider an emergent language framework that uses variational encoders for a disentangled representation of points that reside on a given manifold. Trajectories between points on a manifold can then be described as evolving semantic descriptions. We also show that such probabilistic descriptions (bounded) can be used to improve semantic consistency by constructing defences against adversarial attacks. We evaluate our methods with improved semantic robustness and OoD detection capability, explainable and editable semantic interpolation, and improved classification accuracy under patch attacks. We also discuss the limitation in diffusion models.

Related papers

• DDoS: Diffusion Distribution Similarity for Out-of-Distribution Detection [28.96695036746856]
  Out-of-Distribution (OoD) detection determines whether the given samples are from the training distribution of the classifier-under-protection, i.e., the In-Distribution (InD) or from a different OoD. Recent researches introduce diffusion models pre-trained on InD data to advocate OoD detection by transferring an OoD image into a generated one that is close to InD. Existing diffusion-based detectors adopt perceptual metrics on the two images to measure such disparities, but ignore a fundamental fact: Perceptual metrics are devised essentially for human-perceived similarities of low
  arXiv  Detail & Related papers  (2024-09-16T08:50:47Z)
• Diffusion on the Probability Simplex [24.115365081118604]
  Diffusion models learn to reverse the progressive noising of a data distribution to create a generative model. We propose a method of performing diffusion on the probability simplex. We find that our methodology naturally extends to include diffusion on the unit cube which has applications for bounded image generation.
  arXiv  Detail & Related papers  (2023-09-05T18:52:35Z)
• ChiroDiff: Modelling chirographic data with Diffusion Models [132.5223191478268]
  We introduce a powerful model-class namely "Denoising Diffusion Probabilistic Models" or DDPMs for chirographic data. Our model named "ChiroDiff", being non-autoregressive, learns to capture holistic concepts and therefore remains resilient to higher temporal sampling rate.
  arXiv  Detail & Related papers  (2023-04-07T15:17:48Z)
• Your Diffusion Model is Secretly a Zero-Shot Classifier [90.40799216880342]
  We show that density estimates from large-scale text-to-image diffusion models can be leveraged to perform zero-shot classification. Our generative approach to classification attains strong results on a variety of benchmarks. Our results are a step toward using generative over discriminative models for downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T17:59:56Z)
• Bayesian Deep Learning for Affordance Segmentation in images [3.15834651147911]
  We present a novel Bayesian deep network to detect affordances in images. We quantify the distribution of the aleatoric and epistemic variance at the spatial level. Our results outperform the state-of-the-art of deterministic networks.
  arXiv  Detail & Related papers  (2023-03-02T00:01:13Z)
• Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors [64.24948495708337]
  We introduce a new method that brings predicted samples to the training data manifold using a pretrained unconditional diffusion model. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks.
  arXiv  Detail & Related papers  (2022-12-14T17:26:35Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Sampling Based On Natural Image Statistics Improves Local Surrogate Explainers [111.31448606885672]
  Surrogate explainers are a popular post-hoc interpretability method to further understand how a model arrives at a prediction. We propose two approaches to do so, namely (1) altering the method for sampling the local neighbourhood and (2) using perceptual metrics to convey some of the properties of the distribution of natural images.
  arXiv  Detail & Related papers  (2022-08-08T08:10:13Z)
• Joint Estimation of Image Representations and their Lie Invariants [57.3768308075675]
  Images encode both the state of the world and its content. The automatic extraction of this information is challenging because of the high-dimensionality and entangled encoding inherent to the image representation. This article introduces two theoretical approaches aimed at the resolution of these challenges.
  arXiv  Detail & Related papers  (2020-12-05T00:07:41Z)
• CQ-VAE: Coordinate Quantized VAE for Uncertainty Estimation with Application to Disk Shape Analysis from Lumbar Spine MRI Images [1.5841288368322592]
  We propose a powerful generative model to learn a representation of ambiguity and to generate probabilistic outputs. Our model, named Coordinate Quantization Variational Autoencoder (CQ-VAE), employs a discrete latent space with an internal discrete probability distribution. A matching algorithm is used to establish the correspondence between model-generated samples and "ground-truth" samples.
  arXiv  Detail & Related papers  (2020-10-17T04:25:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.