AptSim2Real: Approximately-Paired Sim-to-Real Image Translation

• URL: http://arxiv.org/abs/2303.12704v2
• Date: Thu, 23 Mar 2023 04:32:57 GMT
• Title: AptSim2Real: Approximately-Paired Sim-to-Real Image Translation
• Authors: Charles Y Zhang and Ashish Shrivastava
• Abstract summary: Sim-to-real transfer modifies simulated images to better match real-world data. AptSim2Real exploits the fact that simulators can generate scenes loosely resembling real-world scenes in terms of lighting, environment, and composition. Our novel training strategy results in significant qualitative and quantitative improvements, with up to a 24% improvement in FID score.
• Score: 8.208569626646035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Advancements in graphics technology has increased the use of simulated data for training machine learning models. However, the simulated data often differs from real-world data, creating a distribution gap that can decrease the efficacy of models trained on simulation data in real-world applications. To mitigate this gap, sim-to-real domain transfer modifies simulated images to better match real-world data, enabling the effective use of simulation data in model training. Sim-to-real transfer utilizes image translation methods, which are divided into two main categories: paired and unpaired image-to-image translation. Paired image translation requires a perfect pixel match, making it difficult to apply in practice due to the lack of pixel-wise correspondence between simulation and real-world data. Unpaired image translation, while more suitable for sim-to-real transfer, is still challenging to learn for complex natural scenes. To address these challenges, we propose a third category: approximately-paired sim-to-real translation, where the source and target images do not need to be exactly paired. Our approximately-paired method, AptSim2Real, exploits the fact that simulators can generate scenes loosely resembling real-world scenes in terms of lighting, environment, and composition. Our novel training strategy results in significant qualitative and quantitative improvements, with up to a 24% improvement in FID score compared to the state-of-the-art unpaired image-translation methods.

Related papers

• Natural Language Can Help Bridge the Sim2Real Gap [9.458180590551715]
  Sim2Real is a promising paradigm for overcoming data scarcity in the real-world target domain. We propose using natural language descriptions of images as a unifying signal across domains. We demonstrate that training the image encoder to predict the language description serves as a useful, data-efficient pretraining step.
  arXiv  Detail & Related papers  (2024-05-16T12:02:02Z)
• Translating Simulation Images to X-ray Images via Multi-Scale Semantic Matching [16.175115921436582]
  We propose a new method to translate simulation images from an endovascular simulator to X-ray images. We apply self-domain semantic matching to ensure that the input image and the generated image have the same positional semantic relationships. Our method generates realistic X-ray images and outperforms other state-of-the-art approaches by a large margin.
  arXiv  Detail & Related papers  (2023-04-16T04:49:46Z)
• Unpaired Translation from Semantic Label Maps to Images by Leveraging Domain-Specific Simulations [11.638139969660266]
  We introduce a contrastive learning framework for generating photorealistic images from simulated label maps. Our proposed method is shown to generate realistic and scene-accurate translations.
  arXiv  Detail & Related papers  (2023-02-21T14:36:18Z)
• Sim2real Transfer Learning for Point Cloud Segmentation: An Industrial Application Case on Autonomous Disassembly [55.41644538483948]
  We present an industrial application case that uses sim2real transfer learning for point cloud data. We provide insights on how to generate and process synthetic point cloud data. A novel patch-based attention network is proposed additionally to tackle this problem.
  arXiv  Detail & Related papers  (2023-01-12T14:00:37Z)
• sim2real: Cardiac MR Image Simulation-to-Real Translation via Unsupervised GANs [0.4433315630787158]
  We provide image simulation on virtual XCAT subjects with varying anatomies. We propose sim2real translation network to improve image realism. Our usability experiments suggest that sim2real data exhibits a good potential to augment training data and boost the performance of a segmentation algorithm.
  arXiv  Detail & Related papers  (2022-08-09T16:06:06Z)
• Synthetic-to-Real Domain Adaptation using Contrastive Unpaired Translation [28.19031441659854]
  We propose a multi-step method to obtain training data without manual annotation effort. From 3D object meshes, we generate images using a modern synthesis pipeline. We utilize a state-of-the-art image-to-image translation method to adapt the synthetic images to the real domain.
  arXiv  Detail & Related papers  (2022-03-17T17:13:23Z)
• Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data [74.66568380558172]
  We study the transferability of pre-trained models based on synthetic data generated by graphics simulators to downstream tasks. We introduce Task2Sim, a unified model mapping downstream task representations to optimal simulation parameters. It learns this mapping by training to find the set of best parameters on a set of "seen" tasks. Once trained, it can then be used to predict best simulation parameters for novel "unseen" tasks in one shot.
  arXiv  Detail & Related papers  (2021-11-30T19:25:27Z)
• RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real [74.45688231140689]
  We introduce the RL-scene consistency loss for image translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. We obtain RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning.
  arXiv  Detail & Related papers  (2020-06-16T08:58:07Z)
• Syn2Real Transfer Learning for Image Deraining using Gaussian Processes [92.15895515035795]
  CNN-based methods for image deraining have achieved excellent performance in terms of reconstruction error as well as visual quality. Due to challenges in obtaining real world fully-labeled image deraining datasets, existing methods are trained only on synthetically generated data. We propose a Gaussian Process-based semi-supervised learning framework which enables the network in learning to derain using synthetic dataset.
  arXiv  Detail & Related papers  (2020-06-10T00:33:18Z)
• Domain Adaptation for Image Dehazing [72.15994735131835]
  Most existing methods train a dehazing model on synthetic hazy images, which are less able to generalize well to real hazy images due to domain shift. We propose a domain adaptation paradigm, which consists of an image translation module and two image dehazing modules. Experimental results on both synthetic and real-world images demonstrate that our model performs favorably against the state-of-the-art dehazing algorithms.
  arXiv  Detail & Related papers  (2020-05-10T13:54:56Z)
• SimAug: Learning Robust Representations from Simulation for Trajectory Prediction [78.91518036949918]
  We propose a novel approach to learn robust representation through augmenting the simulation training data. We show that SimAug achieves promising results on three real-world benchmarks using zero real training data.
  arXiv  Detail & Related papers  (2020-04-04T21:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.