Toward Interactive Modulation for Photo-Realistic Image Restoration
- URL: http://arxiv.org/abs/2105.03085v1
- Date: Fri, 7 May 2021 07:05:56 GMT
- Title: Toward Interactive Modulation for Photo-Realistic Image Restoration
- Authors: Haoming Cai and Jingwen He and Qiao Yu and Chao Dong
- Abstract summary: Modulating image restoration level aims to generate a restored image by altering a factor that represents the restoration strength.
This paper presents a Controllable Unet Generative Adversarial Network (CUGAN) to generate high-frequency textures in the modulation tasks.
- Score: 16.610981587637102
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Modulating image restoration level aims to generate a restored image by
altering a factor that represents the restoration strength. Previous works
mainly focused on optimizing the mean squared reconstruction error, which
brings high reconstruction accuracy but lacks finer texture details. This paper
presents a Controllable Unet Generative Adversarial Network (CUGAN) to generate
high-frequency textures in the modulation tasks. CUGAN consists of two modules
-- base networks and condition networks. The base networks comprise a generator
and a discriminator. In the generator, we realize the interactive control of
restoration levels by tuning the weights of different features from different
scales in the Unet architecture. Moreover, we adaptively modulate the
intermediate features in the discriminator according to the severity of
degradations. The condition networks accept the condition vector (encoded
degradation information) as input, then generate modulation parameters for both
the generator and the discriminator. During testing, users can control the
output effects by tweaking the condition vector. We also provide a smooth
transition between GAN and MSE effects by a simple transition method. Extensive
experiments demonstrate that the proposed CUGAN achieves excellent performance
on image restoration modulation tasks.
Related papers
- Look-Around Before You Leap: High-Frequency Injected Transformer for Image Restoration [46.96362010335177]
In this paper, we propose HIT, a simple yet effective High-frequency Injected Transformer for image restoration.
Specifically, we design a window-wise injection module (WIM), which incorporates abundant high-frequency details into the feature map, to provide reliable references for restoring high-quality images.
In addition, we introduce a spatial enhancement unit (SEU) to preserve essential spatial relationships that may be lost due to the computations carried out across channel dimensions in the BIM.
arXiv Detail & Related papers (2024-03-30T08:05:00Z) - Distance Weighted Trans Network for Image Completion [52.318730994423106]
We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components.
CNNs are used to augment the local texture information of coarse priors.
DWT blocks are used to recover certain coarse textures and coherent visual structures.
arXiv Detail & Related papers (2023-10-11T12:46:11Z) - In-Domain GAN Inversion for Faithful Reconstruction and Editability [132.68255553099834]
We propose in-domain GAN inversion, which consists of a domain-guided domain-regularized and a encoder to regularize the inverted code in the native latent space of the pre-trained GAN model.
We make comprehensive analyses on the effects of the encoder structure, the starting inversion point, as well as the inversion parameter space, and observe the trade-off between the reconstruction quality and the editing property.
arXiv Detail & Related papers (2023-09-25T08:42:06Z) - CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware
Training [112.96224800952724]
We propose cascaded modulation GAN (CM-GAN) to generate plausible image structures when dealing with large holes in complex images.
In each decoder block, global modulation is first applied to perform coarse semantic-aware synthesis structure, then spatial modulation is applied on the output of global modulation to further adjust the feature map in a spatially adaptive fashion.
In addition, we design an object-aware training scheme to prevent the network from hallucinating new objects inside holes, fulfilling the needs of object removal tasks in real-world scenarios.
arXiv Detail & Related papers (2022-03-22T16:13:27Z) - GCFSR: a Generative and Controllable Face Super Resolution Method
Without Facial and GAN Priors [23.74675310360883]
GCFSR is a generative and controllable face SR framework.
It reconstructs images with faithful identity information without any additional priors.
It can outperform state-of-the-art methods for large upscaling factors.
arXiv Detail & Related papers (2022-03-14T17:22:19Z) - CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning.
The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery.
The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
arXiv Detail & Related papers (2021-12-31T04:37:11Z) - High-Fidelity GAN Inversion for Image Attribute Editing [61.966946442222735]
We present a novel high-fidelity generative adversarial network (GAN) inversion framework that enables attribute editing with image-specific details well-preserved.
With a low bit-rate latent code, previous works have difficulties in preserving high-fidelity details in reconstructed and edited images.
We propose a distortion consultation approach that employs a distortion map as a reference for high-fidelity reconstruction.
arXiv Detail & Related papers (2021-09-14T11:23:48Z) - Spatially-Adaptive Image Restoration using Distortion-Guided Networks [51.89245800461537]
We present a learning-based solution for restoring images suffering from spatially-varying degradations.
We propose SPAIR, a network design that harnesses distortion-localization information and dynamically adjusts to difficult regions in the image.
arXiv Detail & Related papers (2021-08-19T11:02:25Z) - Toward a Controllable Disentanglement Network [22.968760397814993]
This paper addresses two crucial problems of learning disentangled image representations, namely controlling the degree of disentanglement during image editing, and balancing the disentanglement strength and the reconstruction quality.
By exploring the real-valued space of the soft target representation, we are able to synthesize novel images with the designated properties.
arXiv Detail & Related papers (2020-01-22T16:54:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.