A Hybrid Approach for Closing the Sim2real Appearance Gap in Game Engine Synthetic Datasets

• URL: http://arxiv.org/abs/2605.02291v1
• Date: Mon, 04 May 2026 07:28:01 GMT
• Title: A Hybrid Approach for Closing the Sim2real Appearance Gap in Game Engine Synthetic Datasets
• Abstract summary: Video game engines have been an important source for generating large volumes of visual synthetic datasets.<n>A notable sim2real appearance gap between the synthetic and the real-world images still remains.<n>We investigate the ability of a state-of-the-art image generation and editing diffusion model to enhance the photorealism of synthetic datasets.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video game engines have been an important source for generating large volumes of visual synthetic datasets for training and evaluating computer vision algorithms that are to be deployed in the real world. While the visual fidelity of modern game engines has been significantly improved with technologies such as ray-tracing, a notable sim2real appearance gap between the synthetic and the real-world images still remains, which limits the utilization of synthetic datasets in real-world applications. In this letter, we investigate the ability of a state-of-the-art image generation and editing diffusion model (FLUX.2-4B Klein) to enhance the photorealism of synthetic datasets and compare its performance against a traditional image-to-image translation model (REGEN). Furthermore, we propose a hybrid approach that combines the strong geometry and material transformations of diffusion-based methods with the distribution-matching capabilities of image-to-image translation techniques. Through experiments, it is demonstrated that REGEN outperforms FLUX.2-4B Klein and that by combining both FLUX.2-4B Klein and REGEN models, better visual realism can be achieved compared to using each model individually, while maintaining semantic consistency. The code is available at: https://github.com/stefanos50/Hybrid-Sim2Real

Related papers

• HyPER-GAN: Hybrid Patch-Based Image-to-Image Translation for Real-Time Photorealism Enhancement [2.478819644330144]
  We propose a lightweight image-to-image translation method based on a U-Net-style generator designed for real-time inference.<n>HyPER-GAN outperforms state-of-the-art lightweight paired image-to-image translation methods in terms of inference latency, visual realism, and semantic robustness.
  arXiv  Detail & Related papers  (2026-03-11T10:05:32Z)
• DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer [62.18680935878919]
  We introduce DiffusionHarmonizer, an online generative enhancement framework that transforms renderings into temporally consistent outputs.<n>At its core is a single-step temporally-conditioned enhancer capable of running in online simulators on a single GPU.
  arXiv  Detail & Related papers  (2026-02-27T15:35:30Z)
• Computer vision training dataset generation for robotic environments using Gaussian splatting [0.0]
  This paper introduces a novel pipeline for generating large-scale, highly realistic, and automatically labeled datasets for computer vision tasks in robotic environments.<n>We leverage 3D Gaussian Splatting (3DGS) to create photorealistic representations of the operational environment and objects.<n>A novel, two-pass rendering technique combines the realism of splats with a shadow map generated from proxy meshes.<n> Pixel-perfect segmentation masks are generated automatically and formatted for direct use with object detection models like YOLO.
  arXiv  Detail & Related papers  (2025-12-15T15:00:17Z)
• RealGen: Photorealistic Text-to-Image Generation via Detector-Guided Rewards [53.25632969696776]
  We propose RealGen, a text-to-image framework for photorealistic image generation.<n>Inspired by adversarial generation, RealGen introduces a "Detector Reward" mechanism, which quantifies artifacts and assesses realism.<n>Experiments demonstrate that RealGen significantly outperforms general models like GPT-Image-1 and Qwen-Image, as well as specialized photorealistic models like FLUX-Krea.
  arXiv  Detail & Related papers  (2025-11-29T12:52:26Z)
• CARLA2Real: a tool for reducing the sim2real appearance gap in CARLA simulator [2.478819644330144]
  We employ a state-of-the-art approach to enhance the photorealism of simulated data, aligning them with the visual characteristics of real-world datasets.<n>Based on this, we developed CARLA2Real, an easy-to-use, publicly available tool (plug-in) for the widely used and open-source CARLA simulator.<n>This tool enhances the output of CARLA in near real-time, achieving a frame rate of 13 FPS, translating it to the visual style and realism of real-world datasets.
  arXiv  Detail & Related papers  (2024-10-23T19:33:30Z)
• FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models [14.596090302381647]
  This paper studies photorealism enhancement of rendered images, leveraging generative power from diffusion models on the controlled basis of rendering. We introduce a novel framework to translate rendered images into their realistic counterparts, which consists of two stages: Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG)
  arXiv  Detail & Related papers  (2024-10-18T12:48:22Z)
• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• Is synthetic data from generative models ready for image recognition? [69.42645602062024]
  We study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks.
  arXiv  Detail & Related papers  (2022-10-14T06:54:24Z)
• Rethinking Blur Synthesis for Deep Real-World Image Deblurring [4.00114307523959]
  We propose a novel realistic blur synthesis pipeline to simulate the camera imaging process. We develop an effective deblurring model that captures non-local dependencies and local context in the feature domain simultaneously. A comprehensive experiment on three real-world datasets shows that the proposed deblurring model performs better than state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-09-28T06:50:16Z)
• DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decoder [73.1010640692609]
  We propose a VQ-VAE architecture model with a diffusion decoder (DiVAE) to work as the reconstructing component in image synthesis. Our model achieves state-of-the-art results and generates more photorealistic images specifically.
  arXiv  Detail & Related papers  (2022-06-01T10:39:12Z)
• Two-shot Spatially-varying BRDF and Shape Estimation [89.29020624201708]
  We propose a novel deep learning architecture with a stage-wise estimation of shape and SVBRDF. We create a large-scale synthetic training dataset with domain-randomized geometry and realistic materials. Experiments on both synthetic and real-world datasets show that our network trained on a synthetic dataset can generalize well to real-world images.
  arXiv  Detail & Related papers  (2020-04-01T12:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.