Inverse Problems Leveraging Pre-trained Contrastive Representations

• URL: http://arxiv.org/abs/2110.07439v1
• Date: Thu, 14 Oct 2021 15:06:30 GMT
• Title: Inverse Problems Leveraging Pre-trained Contrastive Representations
• Authors: Sriram Ravula, Georgios Smyrnis, Matt Jordan, Alexandros G. Dimakis
• Abstract summary: We study a new family of inverse problems for recovering representations of corrupted data. We propose a supervised inversion method that uses a contrastive objective to obtain excellent representations for highly corrupted images. Our method outperforms end-to-end baselines even with a fraction of the labeled data in a wide range of forward operators.
• Score: 88.70821497369785
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study a new family of inverse problems for recovering representations of corrupted data. We assume access to a pre-trained representation learning network R(x) that operates on clean images, like CLIP. The problem is to recover the representation of an image R(x), if we are only given a corrupted version A(x), for some known forward operator A. We propose a supervised inversion method that uses a contrastive objective to obtain excellent representations for highly corrupted images. Using a linear probe on our robust representations, we achieve a higher accuracy than end-to-end supervised baselines when classifying images with various types of distortions, including blurring, additive noise, and random pixel masking. We evaluate on a subset of ImageNet and observe that our method is robust to varying levels of distortion. Our method outperforms end-to-end baselines even with a fraction of the labeled data in a wide range of forward operators.

Related papers

• DeepClean: Integrated Distortion Identification and Algorithm Selection for Rectifying Image Corruptions [1.8024397171920883]
  We propose a two-level sequential planning approach for automated image distortion classification and rectification. The advantage of our approach is its dynamic reconfiguration, conditioned on the input image and generalisability to unseen candidate algorithms at inference time.
  arXiv  Detail & Related papers  (2024-07-23T08:57:11Z)
• Learning to Rank Patches for Unbiased Image Redundancy Reduction [80.93989115541966]
  Images suffer from heavy spatial redundancy because pixels in neighboring regions are spatially correlated. Existing approaches strive to overcome this limitation by reducing less meaningful image regions. We propose a self-supervised framework for image redundancy reduction called Learning to Rank Patches.
  arXiv  Detail & Related papers  (2024-03-31T13:12:41Z)
• Solving General Noisy Inverse Problem via Posterior Sampling: A Policy Gradient Viewpoint [21.22750301965104]
  We leverage a pretrained diffusion generative model to solve a wide range of image inverse tasks without task specific model fine-tuning. To precisely estimate the guidance score function of the input image, we propose Diffusion Policy Gradient (DPG) Experiments show that our method is robust to both Gaussian and Poisson noise degradation on multiple linear and non-linear inverse tasks.
  arXiv  Detail & Related papers  (2024-03-15T16:38:47Z)
• What can we learn about a generated image corrupting its latent representation? [57.1841740328509]
  We investigate the hypothesis that we can predict image quality based on its latent representation in the GANs bottleneck. We achieve this by corrupting the latent representation with noise and generating multiple outputs.
  arXiv  Detail & Related papers  (2022-10-12T14:40:32Z)
• Denoising Diffusion Restoration Models [110.1244240726802]
  Denoising Diffusion Restoration Models (DDRM) is an efficient, unsupervised posterior sampling method. We demonstrate DDRM's versatility on several image datasets for super-resolution, deblurring, inpainting, and colorization.
  arXiv  Detail & Related papers  (2022-01-27T20:19:07Z)
• All You Need is RAW: Defending Against Adversarial Attacks with Camera Image Pipelines [31.043289921613933]
  We propose a model-agnostic adversarial defensive method for image-to-image mapping. The method maps the input RGB images to Bayer RAW space and back to output RGB using a learned camera image signal processing pipeline. As a result, the method generalizes to unseen tasks without additional retraining.
  arXiv  Detail & Related papers  (2021-12-16T21:54:26Z)
• R-MNet: A Perceptual Adversarial Network for Image Inpainting [5.471225956329675]
  We propose a Wasserstein GAN combined with a new reverse mask operator, namely Reverse Masking Network (R-MNet), a perceptual adversarial network for image inpainting. We show that our method is able to generalize to high-resolution inpainting task, and further show more realistic outputs that are plausible to the human visual system.
  arXiv  Detail & Related papers  (2020-08-11T10:58:10Z)
• Fully Unsupervised Diversity Denoising with Convolutional Variational Autoencoders [81.30960319178725]
  We propose DivNoising, a denoising approach based on fully convolutional variational autoencoders (VAEs) First we introduce a principled way of formulating the unsupervised denoising problem within the VAE framework by explicitly incorporating imaging noise models into the decoder. We show that such a noise model can either be measured, bootstrapped from noisy data, or co-learned during training.
  arXiv  Detail & Related papers  (2020-06-10T21:28:13Z)
• Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation [181.08127307338654]
  This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. The deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images.
  arXiv  Detail & Related papers  (2020-03-30T17:45:07Z)
• Reducing the Representation Error of GAN Image Priors Using the Deep Decoder [29.12824512060469]
  We show a method for reducing the representation error of GAN priors by modeling images as the linear combination of a GAN prior and a Deep Decoder. For compressive sensing and image superresolution, our hybrid model exhibits consistently higher PSNRs than both the GAN priors and Deep Decoder separately.
  arXiv  Detail & Related papers  (2020-01-23T18:37:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.