One-shot recognition of any material anywhere using contrastive learning with physics-based rendering

• URL: http://arxiv.org/abs/2212.00648v1
• Date: Thu, 1 Dec 2022 16:49:53 GMT
• Title: One-shot recognition of any material anywhere using contrastive learning with physics-based rendering
• Authors: Manuel S. Drehwald (3), Sagi Eppel (1 and 2 and 4), Jolina Li (2 and 4), Han Hao (2), Alan Aspuru-Guzik (1 and 2) ((1) Vector institute, (2) University of Toronto, (3) Karlsruhe Institute of Technology, (4) Innoviz)
• Abstract summary: We present MatSim: a synthetic dataset, a benchmark, and a method for computer vision based recognition of similarities and transitions between materials and textures. The visual recognition of materials is essential to everything from examining food while cooking to inspecting agriculture, chemistry, and industrial products.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present MatSim: a synthetic dataset, a benchmark, and a method for computer vision based recognition of similarities and transitions between materials and textures, focusing on identifying any material under any conditions using one or a few examples (one-shot learning). The visual recognition of materials is essential to everything from examining food while cooking to inspecting agriculture, chemistry, and industrial products. In this work, we utilize giant repositories used by computer graphics artists to generate a new CGI dataset for material similarity. We use physics-based rendering (PBR) repositories for visual material simulation, assign these materials random 3D objects, and render images with a vast range of backgrounds and illumination conditions (HDRI). We add a gradual transition between materials to support applications with a smooth transition between states (like gradually cooked food). We also render materials inside transparent containers to support beverage and chemistry lab use cases. We then train a contrastive learning network to generate a descriptor that identifies unfamiliar materials using a single image. We also present a new benchmark for a few-shot material recognition that contains a wide range of real-world examples, including the state of a chemical reaction, rotten/fresh fruits, states of food, different types of construction materials, types of ground, and many other use cases involving material states, transitions and subclasses. We show that a network trained on the MatSim synthetic dataset outperforms state-of-the-art models like Clip on the benchmark, despite being tested on material classes that were not seen during training. The dataset, benchmark, code and trained models are available online.

Related papers

• MatSwap: Light-aware material transfers in images [18.37330769828654]
  MatSwap is a method to transfer materials to designated surfaces in an image photorealistically. We learn the relationship between the input material and its appearance within the scene, without the need for explicit UV mapping. Our method seamlessly integrates a desired material into the target location in the photograph while retaining the identity of the scene.
  arXiv  Detail & Related papers  (2025-02-11T18:59:59Z)
• Materialist: Physically Based Editing Using Single-Image Inverse Rendering [50.39048790589746]
  We present a method combining a learning-based approach with progressive differentiable rendering. Our method achieves more realistic light material interactions, accurate shadows, and global illumination. We also propose a method for material transparency editing that operates effectively without requiring full scene geometry.
  arXiv  Detail & Related papers  (2025-01-07T11:52:01Z)
• MaPa: Text-driven Photorealistic Material Painting for 3D Shapes [80.66880375862628]
  This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs. Our framework supports high-quality rendering and provides substantial flexibility in editing.
  arXiv  Detail & Related papers  (2024-04-26T17:54:38Z)
• Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials [108.59709545364395]
  GPT-4V can effectively recognize and describe materials, allowing the construction of a detailed material library. The correctly matched materials are then meticulously applied as reference for the new SVBRDF material generation. Make-it-Real offers a streamlined integration into the 3D content creation workflow.
  arXiv  Detail & Related papers  (2024-04-25T17:59:58Z)
• MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets [63.284244910964475]
  We propose a 3D asset material generation framework to infer underlying material from the 2D semantic prior. Based on such a prior model, we devise a mechanism to parse material in 3D space.
  arXiv  Detail & Related papers  (2024-04-22T07:00:17Z)
• Alchemist: Parametric Control of Material Properties with Diffusion Models [51.63031820280475]
  Our method capitalizes on the generative prior of text-to-image models known for photorealism. We show the potential application of our model to material edited NeRFs.
  arXiv  Detail & Related papers  (2023-12-05T18:58:26Z)
• Material Palette: Extraction of Materials from a Single Image [19.410479434979493]
  We propose a method to extract physically-based rendering (PBR) materials from a single real-world image. We map regions of the image to material concepts using a diffusion model, which allows the sampling of texture images resembling each material in the scene. Second, we benefit from a separate network to decompose the generated textures into Spatially Varying BRDFs.
  arXiv  Detail & Related papers  (2023-11-28T18:59:58Z)
• MatFuse: Controllable Material Generation with Diffusion Models [10.993516790237503]
  MatFuse is a unified approach that harnesses the generative power of diffusion models for creation and editing of 3D materials. Our method integrates multiple sources of conditioning, including color palettes, sketches, text, and pictures, enhancing creative possibilities. We demonstrate the effectiveness of MatFuse under multiple conditioning settings and explore the potential of material editing.
  arXiv  Detail & Related papers  (2023-08-22T12:54:48Z)
• PhotoMat: A Material Generator Learned from Single Flash Photos [37.42765147463852]
  Previous generative models for materials have been trained exclusively on synthetic data. We propose PhotoMat: the first material generator trained exclusively on real photos of material samples captured using a cell phone camera with flash. We show that our generated materials have better visual quality than previous material generators trained on synthetic data.
  arXiv  Detail & Related papers  (2023-05-20T22:27:41Z)
• The joint role of geometry and illumination on material recognition [16.01513204879645]
  We study how the interplay of geometry, illumination, and their spatial frequencies affects human performance on material recognition tasks. We train a deep neural network on material recognition tasks to accurately classify materials.
  arXiv  Detail & Related papers  (2021-01-07T11:29:52Z)
• Fed-Sim: Federated Simulation for Medical Imaging [131.56325440976207]
  We introduce a physics-driven generative approach that consists of two learnable neural modules. We show that our data synthesis framework improves the downstream segmentation performance on several datasets.
  arXiv  Detail & Related papers  (2020-09-01T19:17:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.