Learning Zero-Shot Material States Segmentation, by Implanting Natural Image Patterns in Synthetic Data

• URL: http://arxiv.org/abs/2403.03309v5
• Date: Mon, 10 Jun 2024 01:13:22 GMT
• Title: Learning Zero-Shot Material States Segmentation, by Implanting Natural Image Patterns in Synthetic Data
• Authors: Sagi Eppel, Jolina Li, Manuel Drehwald, Alan Aspuru-Guzik,
• Abstract summary: This work aims to bridge the gap by infusing patterns automatically extracted from real-world images into synthetic data. We present the first comprehensive benchmark for zero-shot material state segmentation. We also share 300,000 extracted textures and SVBRDF/PBR materials to facilitate future generation.
• Score: 0.555174246084229
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Visual recognition of materials and their states is essential for understanding the physical world, from identifying wet regions on surfaces or stains on fabrics to detecting infected areas on plants or minerals in rocks. Collecting data that captures this vast variability is complex due to the scattered and gradual nature of material states. Manually annotating real-world images is constrained by cost and precision, while synthetic data, although accurate and inexpensive, lacks real-world diversity. This work aims to bridge this gap by infusing patterns automatically extracted from real-world images into synthetic data. Hence, patterns collected from natural images are used to generate and map materials into synthetic scenes. This unsupervised approach captures the complexity of the real world while maintaining the precision and scalability of synthetic data. We also present the first comprehensive benchmark for zero-shot material state segmentation, utilizing real-world images across a diverse range of domains, including food, soils, construction, plants, liquids, and more, each appears in various states such as wet, dry, infected, cooked, burned, and many others. The annotation includes partial similarity between regions with similar but not identical materials and hard segmentation of only identical material states. This benchmark eluded top foundation models, exposing the limitations of existing data collection methods. Meanwhile, nets trained on the infused data performed significantly better on this and related tasks. The dataset, code, and trained model are available. We also share 300,000 extracted textures and SVBRDF/PBR materials to facilitate future datasets generation.

Related papers

• On Synthetic Texture Datasets: Challenges, Creation, and Curation [1.9567015559455132]
  We create a dataset of 362,880 texture images that span 56 textures. During the process of generating images, we find that NSFW safety filters in image generation pipelines are highly sensitive to texture.
  arXiv  Detail & Related papers  (2024-09-16T14:02:18Z)
• Intrinsic Image Diffusion for Indoor Single-view Material Estimation [55.276815106443976]
  We present Intrinsic Image Diffusion, a generative model for appearance decomposition of indoor scenes. Given a single input view, we sample multiple possible material explanations represented as albedo, roughness, and metallic maps. Our method produces significantly sharper, more consistent, and more detailed materials, outperforming state-of-the-art methods by $1.5dB$ on PSNR and by $45%$ better FID score on albedo prediction.
  arXiv  Detail & Related papers  (2023-12-19T15:56:19Z)
• Perceptual Artifacts Localization for Image Synthesis Tasks [59.638307505334076]
  We introduce a novel dataset comprising 10,168 generated images, each annotated with per-pixel perceptual artifact labels. A segmentation model, trained on our proposed dataset, effectively localizes artifacts across a range of tasks. We propose an innovative zoom-in inpainting pipeline that seamlessly rectifies perceptual artifacts in the generated images.
  arXiv  Detail & Related papers  (2023-10-09T10:22:08Z)
• SyntheWorld: A Large-Scale Synthetic Dataset for Land Cover Mapping and Building Change Detection [20.985372561774415]
  We present SyntheWorld, a synthetic dataset unparalleled in quality, diversity, and scale. It includes 40,000 images with submeter-level pixels and fine-grained land cover annotations of eight categories. We will release SyntheWorld to facilitate remote sensing image processing research.
  arXiv  Detail & Related papers  (2023-09-05T02:42:41Z)
• Analysis of Training Object Detection Models with Synthetic Data [0.0]
  This paper attempts to provide a holistic overview of how to use synthetic data for object detection. We analyse aspects of generating the data as well as techniques used to train the models. Experiments are validated on real data and benchmarked to models trained on real data.
  arXiv  Detail & Related papers  (2022-11-29T10:21:16Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Dataset of Industrial Metal Objects [1.1125968799758437]
  This dataset contains both real-world and synthetic multi-view RGB images with 6D object pose labels. Real-world data is obtained by recording multi-view images of scenes with varying object shapes, materials, carriers, compositions and lighting conditions. Synthetic data is obtained by carefully simulating real-world conditions and varying them in a controlled and realistic way.
  arXiv  Detail & Related papers  (2022-08-08T10:49:06Z)
• Towards 3D Scene Understanding by Referring Synthetic Models [65.74211112607315]
  Methods typically alleviate on-extensive annotations on real scene scans. We explore how synthetic models rely on real scene categories of synthetic features to a unified feature space. Experiments show that our method achieves the average mAP of 46.08% on the ScanNet S3DIS dataset and 55.49% by learning datasets.
  arXiv  Detail & Related papers  (2022-03-20T13:06:15Z)
• From Synthetic to Real: Image Dehazing Collaborating with Unlabeled Real Data [58.50411487497146]
  We propose a novel image dehazing framework collaborating with unlabeled real data. First, we develop a disentangled image dehazing network (DID-Net), which disentangles the feature representations into three component maps. Then a disentangled-consistency mean-teacher network (DMT-Net) is employed to collaborate unlabeled real data for boosting single image dehazing.
  arXiv  Detail & Related papers  (2021-08-06T04:00:28Z)
• Learning Topology from Synthetic Data for Unsupervised Depth Completion [66.26787962258346]
  We present a method for inferring dense depth maps from images and sparse depth measurements. We learn the association of sparse point clouds with dense natural shapes, and using the image as evidence to validate the predicted depth map.
  arXiv  Detail & Related papers  (2021-06-06T00:21:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.