Learning to simulate complex scenes

• URL: http://arxiv.org/abs/2006.14611v1
• Date: Thu, 25 Jun 2020 17:51:34 GMT
• Title: Learning to simulate complex scenes
• Authors: Zhenfeng Xue, Weijie Mao, Liang Zheng
• Abstract summary: This paper explores content adaptation in the context of semantic segmentation. We propose a scalable discretization-and-relaxation (SDR) approach to optimize the attribute values and obtain a training set of similar content to real-world data. Experiment shows our system can generate reasonable and useful scenes, from which we obtain promising real-world segmentation accuracy.
• Score: 18.51564016785853
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data simulation engines like Unity are becoming an increasingly important data source that allows us to acquire ground truth labels conveniently. Moreover, we can flexibly edit the content of an image in the engine, such as objects (position, orientation) and environments (illumination, occlusion). When using simulated data as training sets, its editable content can be leveraged to mimic the distribution of real-world data, and thus reduce the content difference between the synthetic and real domains. This paper explores content adaptation in the context of semantic segmentation, where the complex street scenes are fully synthesized using 19 classes of virtual objects from a first person driver perspective and controlled by 23 attributes. To optimize the attribute values and obtain a training set of similar content to real-world data, we propose a scalable discretization-and-relaxation (SDR) approach. Under a reinforcement learning framework, we formulate attribute optimization as a random-to-optimized mapping problem using a neural network. Our method has three characteristics. 1) Instead of editing attributes of individual objects, we focus on global attributes that have large influence on the scene structure, such as object density and illumination. 2) Attributes are quantized to discrete values, so as to reduce search space and training complexity. 3) Correlated attributes are jointly optimized in a group, so as to avoid meaningless scene structures and find better convergence points. Experiment shows our system can generate reasonable and useful scenes, from which we obtain promising real-world segmentation accuracy compared with existing synthetic training sets.

Related papers

• Hardness-Aware Scene Synthesis for Semi-Supervised 3D Object Detection [59.33188668341604]
  3D object detection serves as the fundamental task of autonomous driving perception. It is costly to obtain high-quality annotations for point cloud data. We propose a hardness-aware scene synthesis (HASS) method to generate adaptive synthetic scenes.
  arXiv  Detail & Related papers  (2024-05-27T17:59:23Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Towards Scale Consistent Monocular Visual Odometry by Learning from the Virtual World [83.36195426897768]
  We propose VRVO, a novel framework for retrieving the absolute scale from virtual data. We first train a scale-aware disparity network using both monocular real images and stereo virtual data. The resulting scale-consistent disparities are then integrated with a direct VO system.
  arXiv  Detail & Related papers  (2022-03-11T01:51:54Z)
• Attribute Descent: Simulating Object-Centric Datasets on the Content Level and Beyond [17.949962340691673]
  Between synthetic and real, a two-level domain gap exists, involving content level and appearance level. We propose an attribute descent approach that automatically optimize engine attributes to enable synthetic data to approximate real-world data. Experiments on image classification and object re-identification confirm that adapted synthetic data can be effectively used in three scenarios.
  arXiv  Detail & Related papers  (2022-02-28T18:58:05Z)
• Sim2Real Object-Centric Keypoint Detection and Description [40.58367357980036]
  Keypoint detection and description play a central role in computer vision. We propose the object-centric formulation, which requires further identifying which object each interest point belongs to. We develop a sim2real contrastive learning mechanism that can generalize the model trained in simulation to real-world applications.
  arXiv  Detail & Related papers  (2022-02-01T15:00:20Z)
• Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation [98.51313127382937]
  We focus on the use of labels in the synthetic domain alone. Our approach introduces both a way to learn neural-invariant representations and a theoretically inspired view on how to sample the data from the simulator. We showcase our approach on the bird's-eye-view vehicle segmentation task with multi-sensor data.
  arXiv  Detail & Related papers  (2021-11-15T18:37:43Z)
• Synthetic Data Are as Good as the Real for Association Knowledge Learning in Multi-object Tracking [19.772968520292345]
  In this paper, we study whether 3D synthetic data can replace real-world videos for association training. Specifically, we introduce a large-scale synthetic data engine named MOTX, where the motion characteristics of cameras and objects are manually configured to be similar to those in real-world datasets. We show that compared with real data, association knowledge obtained from synthetic data can achieve very similar performance on real-world test sets without domain adaption techniques.
  arXiv  Detail & Related papers  (2021-06-30T14:46:36Z)
• Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation [88.04759848307687]
  In Meta-Sim2, we aim to learn the scene structure in addition to parameters, which is a challenging problem due to its discrete nature. We use Reinforcement Learning to train our model, and design a feature space divergence between our synthesized and target images that is key to successful training. We also show that this leads to downstream improvement in the performance of an object detector trained on our generated dataset as opposed to other baseline simulation methods.
  arXiv  Detail & Related papers  (2020-08-20T17:28:45Z)
• Detection and Segmentation of Custom Objects using High Distraction Photorealistic Synthetic Data [0.5076419064097732]
  We show a straightforward and useful methodology for performing instance segmentation using synthetic data. The goal is to achieve high performance on manually-gathered and annotated real-world data of custom objects. This white-paper provides strong evidence that photorealistic simulated data can be used in practical real world applications.
  arXiv  Detail & Related papers  (2020-07-28T16:33:42Z)
• AI Playground: Unreal Engine-based Data Ablation Tool for Deep Learning [0.0]
  Machine learning requires data, but acquiring and labeling real-world data is challenging, expensive, and time-consuming. We present AI Playground (AIP), an open-source, Unreal Engine-based tool for generating and labeling virtual image data. We trained deep neural networks to predict depth values, surface normals, or object labels and assessed each network's intra- and cross-dataset performance.
  arXiv  Detail & Related papers  (2020-07-13T02:04:39Z)
• iFAN: Image-Instance Full Alignment Networks for Adaptive Object Detection [48.83883375118966]
  iFAN aims to precisely align feature distributions on both image and instance levels. It outperforms state-of-the-art methods with a boost of 10%+ AP over the source-only baseline.
  arXiv  Detail & Related papers  (2020-03-09T13:27:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.