Diffusion-based Holistic Texture Rectification and Synthesis
- URL: http://arxiv.org/abs/2309.14759v1
- Date: Tue, 26 Sep 2023 08:44:46 GMT
- Title: Diffusion-based Holistic Texture Rectification and Synthesis
- Authors: Guoqing Hao, Satoshi Iizuka, Kensho Hara, Edgar Simo-Serra, Hirokatsu
Kataoka, Kazuhiro Fukui
- Abstract summary: Traditional texture synthesis approaches focus on generating textures from pristine samples.
We propose a framework that synthesizes holistic textures from degraded samples in natural images.
- Score: 26.144666226217062
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present a novel framework for rectifying occlusions and distortions in
degraded texture samples from natural images. Traditional texture synthesis
approaches focus on generating textures from pristine samples, which
necessitate meticulous preparation by humans and are often unattainable in most
natural images. These challenges stem from the frequent occlusions and
distortions of texture samples in natural images due to obstructions and
variations in object surface geometry. To address these issues, we propose a
framework that synthesizes holistic textures from degraded samples in natural
images, extending the applicability of exemplar-based texture synthesis
techniques. Our framework utilizes a conditional Latent Diffusion Model (LDM)
with a novel occlusion-aware latent transformer. This latent transformer not
only effectively encodes texture features from partially-observed samples
necessary for the generation process of the LDM, but also explicitly captures
long-range dependencies in samples with large occlusions. To train our model,
we introduce a method for generating synthetic data by applying geometric
transformations and free-form mask generation to clean textures. Experimental
results demonstrate that our framework significantly outperforms existing
methods both quantitatively and quantitatively. Furthermore, we conduct
comprehensive ablation studies to validate the different components of our
proposed framework. Results are corroborated by a perceptual user study which
highlights the efficiency of our proposed approach.
Related papers
- Learning Relighting and Intrinsic Decomposition in Neural Radiance Fields [21.057216351934688]
Our research introduces a method that combines relighting with intrinsic decomposition.
By leveraging light variations in scenes to generate pseudo labels, our method provides guidance for intrinsic decomposition.
We validate our method on both synthetic and real-world datasets, achieving convincing results.
arXiv Detail & Related papers (2024-06-16T21:10:37Z) - Generating Non-Stationary Textures using Self-Rectification [70.91414475376698]
This paper addresses the challenge of example-based non-stationary texture synthesis.
We introduce a novel twostep approach wherein users first modify a reference texture using standard image editing tools.
Our proposed method, termed "self-rectification", automatically refines this target into a coherent, seamless texture.
arXiv Detail & Related papers (2024-01-05T15:07:05Z) - RANRAC: Robust Neural Scene Representations via Random Ray Consensus [12.161889666145127]
RANdom RAy Consensus (RANRAC) is an efficient approach to eliminate the effect of inconsistent data.
We formulate a fuzzy adaption of the RANSAC paradigm, enabling its application to large scale models.
Results indicate significant improvements compared to state-of-the-art robust methods for novel-view synthesis.
arXiv Detail & Related papers (2023-12-15T13:33:09Z) - Relightify: Relightable 3D Faces from a Single Image via Diffusion
Models [86.3927548091627]
We present the first approach to use diffusion models as a prior for highly accurate 3D facial BRDF reconstruction from a single image.
In contrast to existing methods, we directly acquire the observed texture from the input image, thus, resulting in more faithful and consistent estimation.
arXiv Detail & Related papers (2023-05-10T11:57:49Z) - ContraNeRF: Generalizable Neural Radiance Fields for Synthetic-to-real
Novel View Synthesis via Contrastive Learning [102.46382882098847]
We first investigate the effects of synthetic data in synthetic-to-real novel view synthesis.
We propose to introduce geometry-aware contrastive learning to learn multi-view consistent features with geometric constraints.
Our method can render images with higher quality and better fine-grained details, outperforming existing generalizable novel view synthesis methods in terms of PSNR, SSIM, and LPIPS.
arXiv Detail & Related papers (2023-03-20T12:06:14Z) - Person Image Synthesis via Denoising Diffusion Model [116.34633988927429]
We show how denoising diffusion models can be applied for high-fidelity person image synthesis.
Our results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios.
arXiv Detail & Related papers (2022-11-22T18:59:50Z) - SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps [3.504542161036043]
We present SeamlessGAN, a method capable of automatically generating tileable texture maps from a single input exemplar.
In contrast to most existing methods, focused solely on solving the synthesis problem, our work tackles both problems, synthesis and tileability, simultaneously.
arXiv Detail & Related papers (2022-01-13T18:24:26Z) - 3D Human Texture Estimation from a Single Image with Transformers [106.6320286821364]
We propose a Transformer-based framework for 3D human texture estimation from a single image.
We also propose a mask-fusion strategy to combine the advantages of the RGB-based and texture-flow-based models.
arXiv Detail & Related papers (2021-09-06T16:00:20Z) - NITES: A Non-Parametric Interpretable Texture Synthesis Method [41.13585191073405]
A non-parametric interpretable texture synthesis method, called the NITES method, is proposed in this work.
NITES is mathematically transparent and efficient in training and inference.
arXiv Detail & Related papers (2020-09-02T22:52:44Z) - CNN Detection of GAN-Generated Face Images based on Cross-Band
Co-occurrences Analysis [34.41021278275805]
Last-generation GAN models allow to generate synthetic images which are visually indistinguishable from natural ones.
We propose a method for distinguishing GAN-generated from natural images by exploiting inconsistencies among spectral bands.
arXiv Detail & Related papers (2020-07-25T10:55:04Z) - Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image
Decomposition [67.9464567157846]
We propose an autoencoder for joint generation of realistic images from synthetic 3D models while simultaneously decomposing real images into their intrinsic shape and appearance properties.
Our experiments confirm that a joint treatment of rendering and decomposition is indeed beneficial and that our approach outperforms state-of-the-art image-to-image translation baselines both qualitatively and quantitatively.
arXiv Detail & Related papers (2020-06-29T12:53:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.