JoIN: Joint GANs Inversion for Intrinsic Image Decomposition
- URL: http://arxiv.org/abs/2305.11321v2
- Date: Tue, 23 Jan 2024 01:09:46 GMT
- Title: JoIN: Joint GANs Inversion for Intrinsic Image Decomposition
- Authors: Viraj Shah, Svetlana Lazebnik, Julien Philip
- Abstract summary: We propose to solve ill-posed inverse imaging problems using a bank of Generative Adversarial Networks (GAN)
Our method builds on the demonstrated success of GANs to capture complex image distributions.
- Score: 16.02463667910604
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this work, we propose to solve ill-posed inverse imaging problems using a
bank of Generative Adversarial Networks (GAN) as a prior and apply our method
to the case of Intrinsic Image Decomposition for faces and materials. Our
method builds on the demonstrated success of GANs to capture complex image
distributions. At the core of our approach is the idea that the latent space of
a GAN is a well-suited optimization domain to solve inverse problems. Given an
input image, we propose to jointly inverse the latent codes of a set of GANs
and combine their outputs to reproduce the input. Contrary to most GAN
inversion methods which are limited to inverting only a single GAN, we
demonstrate that it is possible to maintain distribution priors while inverting
several GANs jointly. We show that our approach is modular, allowing various
forward imaging models, and that it can successfully decompose both synthetic
and real images.
Related papers
- Compositional Inversion for Stable Diffusion Models [64.79261401944994]
Inversion methods generate personalized images by incorporating concepts of interest provided by user images.
Existing methods often suffer from overfitting issues, where the dominant presence of inverted concepts leads to the absence of other desired concepts.
We propose a method that guides the inversion process towards the core distribution for compositional embeddings.
arXiv Detail & Related papers (2023-12-13T10:57:46Z) - Prompt-tuning latent diffusion models for inverse problems [72.13952857287794]
We propose a new method for solving imaging inverse problems using text-to-image latent diffusion models as general priors.
Our method, called P2L, outperforms both image- and latent-diffusion model-based inverse problem solvers on a variety of tasks, such as super-resolution, deblurring, and inpainting.
arXiv Detail & Related papers (2023-10-02T11:31:48Z) - In-Domain GAN Inversion for Faithful Reconstruction and Editability [132.68255553099834]
We propose in-domain GAN inversion, which consists of a domain-guided domain-regularized and a encoder to regularize the inverted code in the native latent space of the pre-trained GAN model.
We make comprehensive analyses on the effects of the encoder structure, the starting inversion point, as well as the inversion parameter space, and observe the trade-off between the reconstruction quality and the editing property.
arXiv Detail & Related papers (2023-09-25T08:42:06Z) - A Variational Perspective on Solving Inverse Problems with Diffusion
Models [101.831766524264]
Inverse tasks can be formulated as inferring a posterior distribution over data.
This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable.
We propose a variational approach that by design seeks to approximate the true posterior distribution.
arXiv Detail & Related papers (2023-05-07T23:00:47Z) - StyleGAN-induced data-driven regularization for inverse problems [2.5138572116292686]
Recent advances in generative adversarial networks (GANs) have opened up the possibility of generating high-resolution images that were impossible to produce previously.
We develop a framework that utilizes the full potential of a pre-trained StyleGAN2 generator for constructing the prior distribution on the underlying image.
Considering the inverse problems of image inpainting and super-resolution, we demonstrate that the proposed approach is competitive with, and sometimes superior to, state-of-the-art GAN-based image reconstruction methods.
arXiv Detail & Related papers (2021-10-07T22:25:30Z) - GAN Inversion for Out-of-Range Images with Geometric Transformations [22.914126221037222]
We propose BDInvert, a novel GAN inversion approach to semantic editing of out-of-range images.
Our experiments show that BDInvert effectively supports semantic editing of out-of-range images with geometric transformations.
arXiv Detail & Related papers (2021-08-20T04:38:40Z) - GAN Inversion: A Survey [125.62848237531945]
GAN inversion aims to invert a given image back into the latent space of a pretrained GAN model.
GAN inversion plays an essential role in enabling the pretrained GAN models such as StyleGAN and BigGAN to be used for real image editing applications.
arXiv Detail & Related papers (2021-01-14T14:11:00Z) - Multimodal Image-to-Image Translation via Mutual Information Estimation
and Maximization [16.54980086211836]
Multimodal image-to-image translation (I2IT) aims to learn a conditional distribution that explores multiple possible images in the target domain given an input image in the source domain.
Conditional generative adversarial networks (cGANs) are often adopted for modeling such a conditional distribution.
We propose a method that explicitly estimates and maximizes the mutual information between the latent code and the output image in cGANs.
arXiv Detail & Related papers (2020-08-08T14:09:23Z) - Image-to-image Mapping with Many Domains by Sparse Attribute Transfer [71.28847881318013]
Unsupervised image-to-image translation consists of learning a pair of mappings between two domains without known pairwise correspondences between points.
Current convention is to approach this task with cycle-consistent GANs.
We propose an alternate approach that directly restricts the generator to performing a simple sparse transformation in a latent layer.
arXiv Detail & Related papers (2020-06-23T19:52:23Z) - Reducing the Representation Error of GAN Image Priors Using the Deep
Decoder [29.12824512060469]
We show a method for reducing the representation error of GAN priors by modeling images as the linear combination of a GAN prior and a Deep Decoder.
For compressive sensing and image superresolution, our hybrid model exhibits consistently higher PSNRs than both the GAN priors and Deep Decoder separately.
arXiv Detail & Related papers (2020-01-23T18:37:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.