Adaptive Network Combination for Single-Image Reflection Removal: A
Domain Generalization Perspective
- URL: http://arxiv.org/abs/2204.01505v1
- Date: Mon, 4 Apr 2022 14:06:11 GMT
- Title: Adaptive Network Combination for Single-Image Reflection Removal: A
Domain Generalization Perspective
- Authors: Ming Liu, Jianan Pan, Zifei Yan, Wangmeng Zuo, Lei Zhang
- Abstract summary: In this paper, we tackle issues by learning SIRR models from a domain perspective.
For each source set, a specific SIRR model is trained to serve as a domain expert of relevant reflection types.
For images from one source set, we train RTAW to only predict expert-wise weights of other domain experts for improving generalization ability.
Experiments show the appealing performance gain of our AdaNEC on different state-of-the-art SIRR networks.
- Score: 68.37624784559728
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, multiple synthetic and real-world datasets have been built to
facilitate the training of deep single image reflection removal (SIRR) models.
Meanwhile, diverse testing sets are also provided with different types of
reflection and scenes. However, the non-negligible domain gaps between training
and testing sets make it difficult to learn deep models generalizing well to
testing images. The diversity of reflections and scenes further makes it a
mission impossible to learn a single model being effective to all testing sets
and real-world reflections. In this paper, we tackle these issues by learning
SIRR models from a domain generalization perspective. Particularly, for each
source set, a specific SIRR model is trained to serve as a domain expert of
relevant reflection types. For a given reflection-contaminated image, we
present a reflection type-aware weighting (RTAW) module to predict expert-wise
weights. RTAW can then be incorporated with adaptive network combination
(AdaNEC) for handling different reflection types and scenes, i.e., generalizing
to unknown domains. Two representative AdaNEC methods, i.e., output fusion (OF)
and network interpolation (NI), are provided by considering both adaptation
levels and efficiency. For images from one source set, we train RTAW to only
predict expert-wise weights of other domain experts for improving
generalization ability, while the weights of all experts are predicted and
employed during testing. An in-domain expert (IDE) loss is presented for
training RTAW. Extensive experiments show the appealing performance gain of our
AdaNEC on different state-of-the-art SIRR networks. Source code and pre-trained
models will available at https://github.com/csmliu/AdaNEC.
Related papers
- GM-DF: Generalized Multi-Scenario Deepfake Detection [49.072106087564144]
Existing face forgery detection usually follows the paradigm of training models in a single domain.
In this paper, we elaborately investigate the generalization capacity of deepfake detection models when jointly trained on multiple face forgery detection datasets.
arXiv Detail & Related papers (2024-06-28T17:42:08Z) - Reflection Invariance Learning for Few-shot Semantic Segmentation [53.20466630330429]
Few-shot semantic segmentation (FSS) aims to segment objects of unseen classes in query images with only a few annotated support images.
This paper proposes a fresh few-shot segmentation framework to mine the reflection invariance in a multi-view matching manner.
Experiments on both PASCAL-$5textiti$ and COCO-$20textiti$ datasets demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2023-06-01T15:14:58Z) - Mask-Based Modeling for Neural Radiance Fields [20.728248301818912]
In this work, we unveil that 3D implicit representation learning can be significantly improved by mask-based modeling.
We propose MRVM-NeRF, which is a self-supervised pretraining target to predict complete scene representations from partially masked features along each ray.
With this pretraining target, MRVM-NeRF enables better use of correlations across different points and views as the geometry priors.
arXiv Detail & Related papers (2023-04-11T04:12:31Z) - Adaptive adversarial training method for improving multi-scale GAN based
on generalization bound theory [14.562893624131531]
This paper pioneered the introduction of PAC-Bayes generalized bound theory into the training analysis of specific models.
We proposed an adaptive training method which can greatly improve the image manipulation ability of multi-scale GANs.
In particular, for the image super-resolution restoration task, the multi-scale GAN model trained by the proposed method achieves a 100% reduction in natural image quality evaluator (NIQE) and a 60% reduction in root mean squared error (RMSE)
arXiv Detail & Related papers (2022-11-30T07:11:56Z) - Diversified Dynamic Routing for Vision Tasks [36.199659460868496]
We propose a novel architecture where each layer is composed of a set of experts.
In our method, the model is explicitly trained to solve the challenge of finding relevant partitioning of the data.
We conduct several experiments on semantic segmentation on Cityscapes and object detection and instance segmentation on MS-COCO.
arXiv Detail & Related papers (2022-09-26T23:27:51Z) - Two-Stage Single Image Reflection Removal with Reflection-Aware Guidance [78.34235841168031]
We present a novel two-stage network with reflection-aware guidance (RAGNet) for single image reflection removal (SIRR)
RAG can be used (i) to mitigate the effect of reflection from the observation, and (ii) to generate mask in partial convolution for mitigating the effect of deviating from linear combination hypothesis.
Experiments on five commonly used datasets demonstrate the quantitative and qualitative superiority of our RAGNet in comparison to the state-of-the-art SIRR methods.
arXiv Detail & Related papers (2020-12-02T03:14:57Z) - SIR: Self-supervised Image Rectification via Seeing the Same Scene from
Multiple Different Lenses [82.56853587380168]
We propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of the same scene from different lens should be the same.
We leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters.
Our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art methods.
arXiv Detail & Related papers (2020-11-30T08:23:25Z) - Unsupervised Single-Image Reflection Separation Using Perceptual Deep
Image Priors [6.333390830515411]
We propose a novel unsupervised framework for single-image reflection separation.
We optimize the parameters of two cross-coupled deep convolutional networks on a target image to generate two exclusive background and reflection layers.
Our results show that our method significantly outperforms the closest unsupervised method in the literature for removing reflections from single images.
arXiv Detail & Related papers (2020-09-01T21:08:30Z) - Two-shot Spatially-varying BRDF and Shape Estimation [89.29020624201708]
We propose a novel deep learning architecture with a stage-wise estimation of shape and SVBRDF.
We create a large-scale synthetic training dataset with domain-randomized geometry and realistic materials.
Experiments on both synthetic and real-world datasets show that our network trained on a synthetic dataset can generalize well to real-world images.
arXiv Detail & Related papers (2020-04-01T12:56:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.