What can we learn about a generated image corrupting its latent representation?

• URL: http://arxiv.org/abs/2210.06257v1
• Date: Wed, 12 Oct 2022 14:40:32 GMT
• Title: What can we learn about a generated image corrupting its latent representation?
• Authors: Agnieszka Tomczak, Aarushi Gupta, Slobodan Ilic, Nassir Navab, Shadi Albarqouni
• Abstract summary: We investigate the hypothesis that we can predict image quality based on its latent representation in the GANs bottleneck. We achieve this by corrupting the latent representation with noise and generating multiple outputs.
• Score: 57.1841740328509
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative adversarial networks (GANs) offer an effective solution to the image-to-image translation problem, thereby allowing for new possibilities in medical imaging. They can translate images from one imaging modality to another at a low cost. For unpaired datasets, they rely mostly on cycle loss. Despite its effectiveness in learning the underlying data distribution, it can lead to a discrepancy between input and output data. The purpose of this work is to investigate the hypothesis that we can predict image quality based on its latent representation in the GANs bottleneck. We achieve this by corrupting the latent representation with noise and generating multiple outputs. The degree of differences between them is interpreted as the strength of the representation: the more robust the latent representation, the fewer changes in the output image the corruption causes. Our results demonstrate that our proposed method has the ability to i) predict uncertain parts of synthesized images, and ii) identify samples that may not be reliable for downstream tasks, e.g., liver segmentation task.

Related papers

• Data Attribution for Text-to-Image Models by Unlearning Synthesized Images [71.23012718682634]
  The goal of data attribution for text-to-image models is to identify the training images that most influence the generation of a new image. We propose a new approach that efficiently identifies highly-influential images.
  arXiv  Detail & Related papers  (2024-06-13T17:59:44Z)
• Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation [51.37092275604371]
  Multimodal machine translation (MMT) simultaneously takes the source sentence and a relevant image as input for translation. Recent studies suggest utilizing powerful text-to-image generation models to provide image inputs. However, synthetic images generated by these models often follow different distributions compared to authentic images.
  arXiv  Detail & Related papers  (2023-10-20T09:06:30Z)
• Chili Pepper Disease Diagnosis via Image Reconstruction Using GrabCut and Generative Adversarial Serial Autoencoder [0.0]
  Disease diagnosis belongs to the field of anomaly detection and aims to distinguish whether plants or fruits are normal or abnormal. Due to the limitation of the performance of image generation, SOTA's methods propose a score calculation method using a latent vector error. We propose a method of generating meaningful images using the GAN structure and classifying three results simultaneously by one discriminator.
  arXiv  Detail & Related papers  (2023-06-21T06:58:46Z)
• Retinal Image Restoration using Transformer and Cycle-Consistent Generative Adversarial Network [0.7868449549351486]
  Medical imaging plays a significant role in detecting and treating various diseases. We propose a retinal image enhancement method using a vision transformer and convolutional neural network.
  arXiv  Detail & Related papers  (2023-03-03T14:10:47Z)
• DEff-GAN: Diverse Attribute Transfer for Few-Shot Image Synthesis [0.38073142980733]
  We extend the single-image GAN method to model multiple images for sample synthesis. Our Data-Efficient GAN (DEff-GAN) generates excellent results when similarities and correspondences can be drawn between the input images or classes.
  arXiv  Detail & Related papers  (2023-02-28T12:43:52Z)
• Causal Transportability for Visual Recognition [70.13627281087325]
  We show that standard classifiers fail because the association between images and labels is not transportable across settings. We then show that the causal effect, which severs all sources of confounding, remains invariant across domains. This motivates us to develop an algorithm to estimate the causal effect for image classification.
  arXiv  Detail & Related papers  (2022-04-26T15:02:11Z)
• Inverse Problems Leveraging Pre-trained Contrastive Representations [88.70821497369785]
  We study a new family of inverse problems for recovering representations of corrupted data. We propose a supervised inversion method that uses a contrastive objective to obtain excellent representations for highly corrupted images. Our method outperforms end-to-end baselines even with a fraction of the labeled data in a wide range of forward operators.
  arXiv  Detail & Related papers  (2021-10-14T15:06:30Z)
• USIS: Unsupervised Semantic Image Synthesis [9.613134538472801]
  We propose a new Unsupervised paradigm for Semantic Image Synthesis (USIS) USIS learns to output images with visually separable semantic classes using a self-supervised segmentation loss. In order to match the color and texture distribution of real images without losing high-frequency information, we propose to use whole image wavelet-based discrimination.
  arXiv  Detail & Related papers  (2021-09-29T20:48:41Z)
• Adversarial Semantic Data Augmentation for Human Pose Estimation [96.75411357541438]
  We propose Semantic Data Augmentation (SDA), a method that augments images by pasting segmented body parts with various semantic granularity. We also propose Adversarial Semantic Data Augmentation (ASDA), which exploits a generative network to dynamiclly predict tailored pasting configuration. State-of-the-art results are achieved on challenging benchmarks.
  arXiv  Detail & Related papers  (2020-08-03T07:56:04Z)
• ContraGAN: Contrastive Learning for Conditional Image Generation [14.077997868828177]
  Conditional Generative Adversarial Networks (GAN) are used to generate diverse images using class label information. We propose ContraGAN that considers relations between multiple image embeddings in the same batch (data-to-data relations) as well as the data-to-class relations by using a conditional contrastive loss. The experimental results show that ContraGAN outperforms state-of-the-art-models by 7.3% and 7.7% on Tiny ImageNet and ImageNet datasets, respectively.
  arXiv  Detail & Related papers  (2020-06-23T00:49:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.