ZeST: Zero-Shot Material Transfer from a Single Image

• URL: http://arxiv.org/abs/2404.06425v1
• Date: Tue, 9 Apr 2024 16:15:03 GMT
• Title: ZeST: Zero-Shot Material Transfer from a Single Image
• Authors: Ta-Ying Cheng, Prafull Sharma, Andrew Markham, Niki Trigoni, Varun Jampani,
• Abstract summary: ZeST is a method for zero-shot material transfer to an object in the input image given a material exemplar image. We show the application of ZeST to perform multiple edits and robust material assignment under different illuminations.
• Score: 59.714441587735614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose ZeST, a method for zero-shot material transfer to an object in the input image given a material exemplar image. ZeST leverages existing diffusion adapters to extract implicit material representation from the exemplar image. This representation is used to transfer the material using pre-trained inpainting diffusion model on the object in the input image using depth estimates as geometry cue and grayscale object shading as illumination cues. The method works on real images without any training resulting a zero-shot approach. Both qualitative and quantitative results on real and synthetic datasets demonstrate that ZeST outputs photorealistic images with transferred materials. We also show the application of ZeST to perform multiple edits and robust material assignment under different illuminations. Project Page: https://ttchengab.github.io/zest

Related papers

• MatSwap: Light-aware material transfers in images [18.37330769828654]
  MatSwap is a method to transfer materials to designated surfaces in an image photorealistically. We learn the relationship between the input material and its appearance within the scene, without the need for explicit UV mapping. Our method seamlessly integrates a desired material into the target location in the photograph while retaining the identity of the scene.
  arXiv  Detail & Related papers  (2025-02-11T18:59:59Z)
• Materialist: Physically Based Editing Using Single-Image Inverse Rendering [50.39048790589746]
  We present a method combining a learning-based approach with progressive differentiable rendering. Our method achieves more realistic light material interactions, accurate shadows, and global illumination. We also propose a method for material transparency editing that operates effectively without requiring full scene geometry.
  arXiv  Detail & Related papers  (2025-01-07T11:52:01Z)
• IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations [64.07859467542664]
  Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics. Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs. We introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations.
  arXiv  Detail & Related papers  (2024-12-16T18:52:56Z)
• MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets [63.284244910964475]
  We propose a 3D asset material generation framework to infer underlying material from the 2D semantic prior. Based on such a prior model, we devise a mechanism to parse material in 3D space.
  arXiv  Detail & Related papers  (2024-04-22T07:00:17Z)
• IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination [37.96484120807323]
  This paper aims to recover object materials from posed images captured under an unknown static lighting condition. We learn the material prior with a generative model for regularizing the optimization process. Experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery.
  arXiv  Detail & Related papers  (2024-04-17T17:45:08Z)
• Material Palette: Extraction of Materials from a Single Image [19.410479434979493]
  We propose a method to extract physically-based rendering (PBR) materials from a single real-world image. We map regions of the image to material concepts using a diffusion model, which allows the sampling of texture images resembling each material in the scene. Second, we benefit from a separate network to decompose the generated textures into Spatially Varying BRDFs.
  arXiv  Detail & Related papers  (2023-11-28T18:59:58Z)
• NDJIR: Neural Direct and Joint Inverse Rendering for Geometry, Lights, and Materials of Real Object [5.665283675533071]
  We propose neural direct and joint inverse rendering, NDJIR. Our proposed method can decompose semantically well for real object in photogrammetric setting.
  arXiv  Detail & Related papers  (2023-02-02T13:21:03Z)
• Designing An Illumination-Aware Network for Deep Image Relighting [69.750906769976]
  We present an Illumination-Aware Network (IAN) which follows the guidance from hierarchical sampling to progressively relight a scene from a single image. In addition, an Illumination-Aware Residual Block (IARB) is designed to approximate the physical rendering process. Experimental results show that our proposed method produces better quantitative and qualitative relighting results than previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-07-21T16:21:24Z)
• Enhancing Low-Light Images in Real World via Cross-Image Disentanglement [58.754943762945864]
  We propose a new low-light image enhancement dataset consisting of misaligned training images with real-world corruptions. Our model achieves state-of-the-art performances on both the newly proposed dataset and other popular low-light datasets.
  arXiv  Detail & Related papers  (2022-01-10T03:12:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.