Unsupervised Synthetic Image Refinement via Contrastive Learning and Consistent Semantic-Structural Constraints

• URL: http://arxiv.org/abs/2304.12591v2
• Date: Wed, 26 Apr 2023 06:03:25 GMT
• Title: Unsupervised Synthetic Image Refinement via Contrastive Learning and Consistent Semantic-Structural Constraints
• Authors: Ganning Zhao, Tingwei Shen, Suya You, and C.-C. Jay Kuo
• Abstract summary: Contrastive learning (CL) has been successfully used to pull correlated patches together and push uncorrelated ones apart. In this work, we exploit semantic and structural consistency between synthetic and refined images and adopt CL to reduce the semantic distortion.
• Score: 32.07631215590755
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ensuring the realism of computer-generated synthetic images is crucial to deep neural network (DNN) training. Due to different semantic distributions between synthetic and real-world captured datasets, there exists semantic mismatch between synthetic and refined images, which in turn results in the semantic distortion. Recently, contrastive learning (CL) has been successfully used to pull correlated patches together and push uncorrelated ones apart. In this work, we exploit semantic and structural consistency between synthetic and refined images and adopt CL to reduce the semantic distortion. Besides, we incorporate hard negative mining to improve the performance furthermore. We compare the performance of our method with several other benchmarking methods using qualitative and quantitative measures and show that our method offers the state-of-the-art performance.

Related papers

• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• Semantic Ensemble Loss and Latent Refinement for High-Fidelity Neural Image Compression [58.618625678054826]
  This study presents an enhanced neural compression method designed for optimal visual fidelity. We have trained our model with a sophisticated semantic ensemble loss, integrating Charbonnier loss, perceptual loss, style loss, and a non-binary adversarial loss. Our empirical findings demonstrate that this approach significantly improves the statistical fidelity of neural image compression.
  arXiv  Detail & Related papers  (2024-01-25T08:11:27Z)
• The Right Losses for the Right Gains: Improving the Semantic Consistency of Deep Text-to-Image Generation with Distribution-Sensitive Losses [0.35898124827270983]
  We propose a contrastive learning approach with a novel combination of two loss functions: fake-to-fake loss and fake-to-real loss. We test this approach on two baseline models: SSAGAN and AttnGAN. Results show that our approach improves the qualitative results on AttnGAN with style blocks on the CUB dataset.
  arXiv  Detail & Related papers  (2023-12-18T00:05:28Z)
• Robustness-Guided Image Synthesis for Data-Free Quantization [15.91924736452861]
  We propose Robustness-Guided Image Synthesis (RIS) to enrich the semantics of synthetic images and improve image diversity. RIS is a simple but effective method to enrich the semantics of synthetic images and improve image diversity. We achieve state-of-the-art performance for various settings on data-free quantization and can be extended to other data-free compression tasks.
  arXiv  Detail & Related papers  (2023-10-05T16:39:14Z)
• Bridging Synthetic and Real Images: a Transferable and Multiple Consistency aided Fundus Image Enhancement Framework [61.74188977009786]
  We propose an end-to-end optimized teacher-student framework to simultaneously conduct image enhancement and domain adaptation. We also propose a novel multi-stage multi-attention guided enhancement network (MAGE-Net) as the backbones of our teacher and student network.
  arXiv  Detail & Related papers  (2023-02-23T06:16:15Z)
• Contrastive Image Synthesis and Self-supervised Feature Adaptation for Cross-Modality Biomedical Image Segmentation [8.772764547425291]
  CISFA builds on image domain translation and unsupervised feature adaptation for cross-modality biomedical image segmentation. We use a one-sided generative model and add a weighted patch-wise contrastive loss between sampled patches of the input image and the corresponding synthetic image. We evaluate our methods on segmentation tasks containing CT and MRI images for abdominal cavities and whole hearts.
  arXiv  Detail & Related papers  (2022-07-27T01:49:26Z)
• Deep Semantic Statistics Matching (D2SM) Denoising Network [70.01091467628068]
  We introduce the Deep Semantic Statistics Matching (D2SM) Denoising Network. It exploits semantic features of pretrained classification networks, then it implicitly matches the probabilistic distribution of clear images at the semantic feature space. By learning to preserve the semantic distribution of denoised images, we empirically find our method significantly improves the denoising capabilities of networks.
  arXiv  Detail & Related papers  (2022-07-19T14:35:42Z)
• Semantic Image Synthesis via Diffusion Models [159.4285444680301]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks. Recent work on semantic image synthesis mainly follows the emphde facto Generative Adversarial Nets (GANs)
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• KLIEP-based Density Ratio Estimation for Semantically Consistent Synthetic to Real Images Adaptation in Urban Traffic Scenes [46.526831127902604]
  We show how adversarial training alone can introduce semantic inconsistencies in translated images. We propose density prematching strategy using KLIEP-based density ratio estimation procedure.
  arXiv  Detail & Related papers  (2021-05-26T13:59:19Z)
• You Only Need Adversarial Supervision for Semantic Image Synthesis [84.83711654797342]
  We propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We show that images synthesized by our model are more diverse and follow the color and texture of real images more closely.
  arXiv  Detail & Related papers  (2020-12-08T23:00:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.