SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes

• URL: http://arxiv.org/abs/2307.07333v2
• Date: Fri, 23 Feb 2024 14:34:18 GMT
• Title: SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes
• Authors: Zhili Ng, Haozhe Wang, Zhengshen Zhang, Francis Tay Eng Hock, and Marcelo H. Ang Jr
• Abstract summary: We present SynTable, a Python-based dataset generator built using NVIDIA's Isaac Sim Replicator Composer. Our dataset generation tool can render a complex 3D scene containing object meshes, materials, textures, lighting, and backgrounds. We demonstrate the use of a sample dataset generated using SynTable by ray tracing for training a state-of-the-art model, UOAIS-Net.
• Score: 2.8661021832561757
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we present SynTable, a unified and flexible Python-based dataset generator built using NVIDIA's Isaac Sim Replicator Composer for generating high-quality synthetic datasets for unseen object amodal instance segmentation of cluttered tabletop scenes. Our dataset generation tool can render a complex 3D scene containing object meshes, materials, textures, lighting, and backgrounds. Metadata, such as modal and amodal instance segmentation masks, occlusion masks, depth maps, bounding boxes, and material properties, can be generated to automatically annotate the scene according to the users' requirements. Our tool eliminates the need for manual labeling in the dataset generation process while ensuring the quality and accuracy of the dataset. In this work, we discuss our design goals, framework architecture, and the performance of our tool. We demonstrate the use of a sample dataset generated using SynTable by ray tracing for training a state-of-the-art model, UOAIS-Net. The results show significantly improved performance in Sim-to-Real transfer when evaluated on the OSD-Amodal dataset. We offer this tool as an open-source, easy-to-use, photorealistic dataset generator for advancing research in deep learning and synthetic data generation.

Related papers

• MASSTAR: A Multi-Modal and Large-Scale Scene Dataset with a Versatile Toolchain for Surface Prediction and Completion [25.44529512862336]
  MASSTAR is a multi-modal lArge-scale scene dataset with a verSatile Toolchain for surfAce pRediction and completion. We develop a versatile and efficient toolchain for processing the raw 3D data from the environments. We generate an example dataset composed of over a thousand scene-level models with partial real-world data.
  arXiv  Detail & Related papers  (2024-03-18T11:35:18Z)
• View-Dependent Octree-based Mesh Extraction in Unbounded Scenes for Procedural Synthetic Data [71.22495169640239]
  Procedural signed distance functions (SDFs) are a powerful tool for modeling large-scale detailed scenes. We propose OcMesher, a mesh extraction algorithm that efficiently handles high-detail unbounded scenes with perfect view-consistency.
  arXiv  Detail & Related papers  (2023-12-13T18:56:13Z)
• UniG3D: A Unified 3D Object Generation Dataset [75.49544172927749]
  UniG3D is a unified 3D object generation dataset constructed by employing a universal data transformation pipeline on ShapeNet datasets. This pipeline converts each raw 3D model into comprehensive multi-modal data representation. The selection of data sources for our dataset is based on their scale and quality.
  arXiv  Detail & Related papers  (2023-06-19T07:03:45Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis [104.53930611219654]
  We present a large-scale synthetic dataset for novel view synthesis consisting of 300k images rendered from nearly 2000 complex scenes. The dataset is orders of magnitude larger than existing synthetic datasets for novel view synthesis. Using 4 distinct sources of high-quality 3D meshes, the scenes of our dataset exhibit challenging variations in camera views, lighting, shape, materials, and textures.
  arXiv  Detail & Related papers  (2022-05-14T13:15:32Z)
• Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans [103.92680099373567]
  This paper introduces a pipeline to parametrically sample and render multi-task vision datasets from comprehensive 3D scans from the real world. Changing the sampling parameters allows one to "steer" the generated datasets to emphasize specific information. Common architectures trained on a generated starter dataset reached state-of-the-art performance on multiple common vision tasks and benchmarks.
  arXiv  Detail & Related papers  (2021-10-11T04:21:46Z)
• SynPick: A Dataset for Dynamic Bin Picking Scene Understanding [25.706613724135046]
  We present SynPick, a synthetic dataset for dynamic scene understanding in binpicking scenarios. In contrast to existing datasets, our dataset is both situated in a realistic industrial application domain. The dataset is compatible with the popular BOP dataset format.
  arXiv  Detail & Related papers  (2021-07-10T14:58:43Z)
• Unity Perception: Generate Synthetic Data for Computer Vision [9.479256069071315]
  We introduce the Unity Perception package which aims to simplify and accelerate the process of generating synthetic datasets for computer vision tasks. This open-source package extends the Unity Editor and engine components to generate perfectly annotated examples for several common computer vision tasks. We provide an overview of the provided tools and how they work, and demonstrate the value of the generated synthetic datasets by training a 2D object detection model.
  arXiv  Detail & Related papers  (2021-07-09T07:09:00Z)
• Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation [88.04759848307687]
  In Meta-Sim2, we aim to learn the scene structure in addition to parameters, which is a challenging problem due to its discrete nature. We use Reinforcement Learning to train our model, and design a feature space divergence between our synthesized and target images that is key to successful training. We also show that this leads to downstream improvement in the performance of an object detector trained on our generated dataset as opposed to other baseline simulation methods.
  arXiv  Detail & Related papers  (2020-08-20T17:28:45Z)
• Detection and Segmentation of Custom Objects using High Distraction Photorealistic Synthetic Data [0.5076419064097732]
  We show a straightforward and useful methodology for performing instance segmentation using synthetic data. The goal is to achieve high performance on manually-gathered and annotated real-world data of custom objects. This white-paper provides strong evidence that photorealistic simulated data can be used in practical real world applications.
  arXiv  Detail & Related papers  (2020-07-28T16:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.