Synthetica: Large Scale Synthetic Data for Robot Perception

• URL: http://arxiv.org/abs/2410.21153v1
• Date: Mon, 28 Oct 2024 15:50:56 GMT
• Title: Synthetica: Large Scale Synthetic Data for Robot Perception
• Authors: Ritvik Singh, Jingzhou Liu, Karl Van Wyk, Yu-Wei Chao, Jean-Francois Lafleche, Florian Shkurti, Nathan Ratliff, Ankur Handa,
• Abstract summary: We present Synthetica, a method for large-scale synthetic data generation for training robust state estimators. This paper focuses on the task of object detection, an important problem which can serve as the front-end for most state estimation problems. We leverage data from a ray-tracing, generating 2.7 million images, to train highly accurate real-time detection transformers. We demonstrate state-of-the-art performance on the task of object detection while having detectors that run at 50-100Hz which is 9 times faster than the prior SOTA.
• Score: 21.415878105900187
• License:
• Abstract: Vision-based object detectors are a crucial basis for robotics applications as they provide valuable information about object localisation in the environment. These need to ensure high reliability in different lighting conditions, occlusions, and visual artifacts, all while running in real-time. Collecting and annotating real-world data for these networks is prohibitively time consuming and costly, especially for custom assets, such as industrial objects, making it untenable for generalization to in-the-wild scenarios. To this end, we present Synthetica, a method for large-scale synthetic data generation for training robust state estimators. This paper focuses on the task of object detection, an important problem which can serve as the front-end for most state estimation problems, such as pose estimation. Leveraging data from a photorealistic ray-tracing renderer, we scale up data generation, generating 2.7 million images, to train highly accurate real-time detection transformers. We present a collection of rendering randomization and training-time data augmentation techniques conducive to robust sim-to-real performance for vision tasks. We demonstrate state-of-the-art performance on the task of object detection while having detectors that run at 50-100Hz which is 9 times faster than the prior SOTA. We further demonstrate the usefulness of our training methodology for robotics applications by showcasing a pipeline for use in the real world with custom objects for which there do not exist prior datasets. Our work highlights the importance of scaling synthetic data generation for robust sim-to-real transfer while achieving the fastest real-time inference speeds. Videos and supplementary information can be found at this URL: https://sites.google.com/view/synthetica-vision.

Related papers

• Learning from synthetic data generated with GRADE [0.6982738885923204]
  We present a framework for generating realistic animated dynamic environments (GRADE) for robotics research. GRADE supports full simulation control, ROS integration, realistic physics, while being in an engine that produces high visual fidelity images and ground truth data. We show that, even training using only synthetic data, can generalize well to real-world images in the same application domain.
  arXiv  Detail & Related papers  (2023-05-07T14:13:04Z)
• Synthetic Data for Object Classification in Industrial Applications [53.180678723280145]
  In object classification, capturing a large number of images per object and in different conditions is not always possible. This work explores the creation of artificial images using a game engine to cope with limited data in the training dataset.
  arXiv  Detail & Related papers  (2022-12-09T11:43:04Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Neural-Sim: Learning to Generate Training Data with NeRF [31.81496344354997]
  We present the first fully differentiable synthetic data pipeline that uses Neural Radiance Fields (NeRFs) in a closed-loop with a target application's loss function. Our approach generates data on-demand, with no human labor, to maximize accuracy for a target task.
  arXiv  Detail & Related papers  (2022-07-22T22:48:33Z)
• Hands-Up: Leveraging Synthetic Data for Hands-On-Wheel Detection [0.38233569758620045]
  This work demonstrates the use of synthetic photo-realistic in-cabin data to train a Driver Monitoring System. We show how performing error analysis and generating the missing edge-cases in our platform boosts performance. This showcases the ability of human-centric synthetic data to generalize well to the real world.
  arXiv  Detail & Related papers  (2022-05-31T23:34:12Z)
• MetaGraspNet: A Large-Scale Benchmark Dataset for Vision-driven Robotic Grasping via Physics-based Metaverse Synthesis [78.26022688167133]
  We present a large-scale benchmark dataset for vision-driven robotic grasping via physics-based metaverse synthesis. The proposed dataset contains 100,000 images and 25 different object types. We also propose a new layout-weighted performance metric alongside the dataset for evaluating object detection and segmentation performance.
  arXiv  Detail & Related papers  (2021-12-29T17:23:24Z)
• RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection [138.2892824662943]
  A promising solution is to make better use of the synthetic dataset, which consists of CAD object models, to boost the learning on real datasets. Recent work on 3D pre-training exhibits failure when transfer features learned on synthetic objects to other real-world applications. In this work, we put forward a new method called RandomRooms to accomplish this objective.
  arXiv  Detail & Related papers  (2021-08-17T17:56:12Z)
• UnrealROX+: An Improved Tool for Acquiring Synthetic Data from Virtual 3D Environments [14.453602631430508]
  We present an improved version of UnrealROX, a tool to generate synthetic data from robotic images. Un UnrealROX+ includes new features such as generating albedo or a Python API for interacting with the virtual environment from Deep Learning frameworks.
  arXiv  Detail & Related papers  (2021-04-23T18:45:42Z)
• Object-based Illumination Estimation with Rendering-aware Neural Networks [56.01734918693844]
  We present a scheme for fast environment light estimation from the RGBD appearance of individual objects and their local image areas. With the estimated lighting, virtual objects can be rendered in AR scenarios with shading that is consistent to the real scene.
  arXiv  Detail & Related papers  (2020-08-06T08:23:19Z)
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation [62.22058066456076]
  Reinforcement Learning (RL) represents powerful tools to solve complex robotic tasks. RL does not work directly in the real-world, which is known as the sim-to-real transfer problem. We propose a method that learns on an observation space constructed by point clouds and environment randomization.
  arXiv  Detail & Related papers  (2020-07-27T17:46:59Z)
• Deflating Dataset Bias Using Synthetic Data Augmentation [8.509201763744246]
  State-of-the-art methods for most vision tasks for Autonomous Vehicles (AVs) rely on supervised learning. The goal of this paper is to investigate the use of targeted synthetic data augmentation for filling gaps in real datasets for vision tasks. Empirical studies on three different computer vision tasks of practical use to AVs consistently show that having synthetic data in the training mix provides a significant boost in cross-dataset generalization performance.
  arXiv  Detail & Related papers  (2020-04-28T21:56:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.