Leveraging in-domain supervision for unsupervised image-to-image
translation tasks via multi-stream generators
- URL: http://arxiv.org/abs/2112.15091v1
- Date: Thu, 30 Dec 2021 15:29:36 GMT
- Title: Leveraging in-domain supervision for unsupervised image-to-image
translation tasks via multi-stream generators
- Authors: Dvir Yerushalmi, Dov Danon, Amit H. Bermano
- Abstract summary: We introduce two techniques to incorporate this invaluable in-domain prior knowledge for the benefit of translation quality.
We propose splitting the input data according to semantic masks, explicitly guiding the network to different behavior for the different regions of the image.
In addition, we propose training a semantic segmentation network along with the translation task, and to leverage this output as a loss term that improves robustness.
- Score: 4.726777092009554
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Supervision for image-to-image translation (I2I) tasks is hard to come by,
but bears significant effect on the resulting quality. In this paper, we
observe that for many Unsupervised I2I (UI2I) scenarios, one domain is more
familiar than the other, and offers in-domain prior knowledge, such as semantic
segmentation. We argue that for complex scenes, figuring out the semantic
structure of the domain is hard, especially with no supervision, but is an
important part of a successful I2I operation. We hence introduce two techniques
to incorporate this invaluable in-domain prior knowledge for the benefit of
translation quality: through a novel Multi-Stream generator architecture, and
through a semantic segmentation-based regularization loss term. In essence, we
propose splitting the input data according to semantic masks, explicitly
guiding the network to different behavior for the different regions of the
image. In addition, we propose training a semantic segmentation network along
with the translation task, and to leverage this output as a loss term that
improves robustness. We validate our approach on urban data, demonstrating
superior quality in the challenging UI2I tasks of converting day images to
night ones. In addition, we also demonstrate how reinforcing the target dataset
with our augmented images improves the training of downstream tasks such as the
classical detection one.
Related papers
- Unsupervised Image-to-Image Translation with Generative Prior [103.54337984566877]
Unsupervised image-to-image translation aims to learn the translation between two visual domains without paired data.
We present a novel framework, Generative Prior-guided UN Image-to-image Translation (GP-UNIT), to improve the overall quality and applicability of the translation algorithm.
arXiv Detail & Related papers (2022-04-07T17:59:23Z) - Multi-domain Unsupervised Image-to-Image Translation with Appearance
Adaptive Convolution [62.4972011636884]
We propose a novel multi-domain unsupervised image-to-image translation (MDUIT) framework.
We exploit the decomposed content feature and appearance adaptive convolution to translate an image into a target appearance.
We show that the proposed method produces visually diverse and plausible results in multiple domains compared to the state-of-the-art methods.
arXiv Detail & Related papers (2022-02-06T14:12:34Z) - HYLDA: End-to-end Hybrid Learning Domain Adaptation for LiDAR Semantic
Segmentation [13.87939140266266]
This paper addresses the problem of training a LiDAR semantic segmentation network using a fully-labeled source dataset and a target dataset that only has a small number of labels.
We develop a novel image-to-image translation engine, and couple it with a LiDAR semantic segmentation network, resulting in an integrated domain adaptation architecture we call HYLDA.
arXiv Detail & Related papers (2022-01-14T18:13:09Z) - Self-Supervised Learning of Domain Invariant Features for Depth
Estimation [35.74969527929284]
We tackle the problem of unsupervised synthetic-to-realistic domain adaptation for single image depth estimation.
An essential building block of single image depth estimation is an encoder-decoder task network that takes RGB images as input and produces depth maps as output.
We propose a novel training strategy to force the task network to learn domain invariant representations in a self-supervised manner.
arXiv Detail & Related papers (2021-06-04T16:45:48Z) - Unsupervised Image-to-Image Translation via Pre-trained StyleGAN2
Network [73.5062435623908]
We propose a new I2I translation method that generates a new model in the target domain via a series of model transformations.
By feeding the latent vector into the generated model, we can perform I2I translation between the source domain and target domain.
arXiv Detail & Related papers (2020-10-12T13:51:40Z) - Crossing-Domain Generative Adversarial Networks for Unsupervised
Multi-Domain Image-to-Image Translation [12.692904507625036]
We propose a general framework for unsupervised image-to-image translation across multiple domains.
Our proposed framework consists of a pair of encoders along with a pair of GANs which learns high-level features across different domains to generate diverse and realistic samples from.
arXiv Detail & Related papers (2020-08-27T01:54:07Z) - TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired
Images [102.4003329297039]
An unsupervised image-to-image translation (UI2I) task deals with learning a mapping between two domains without paired images.
We propose TuiGAN, a generative model that is trained on only two unpaired images and amounts to one-shot unsupervised learning.
arXiv Detail & Related papers (2020-04-09T16:23:59Z) - Structured Domain Adaptation with Online Relation Regularization for
Unsupervised Person Re-ID [62.90727103061876]
Unsupervised domain adaptation (UDA) aims at adapting the model trained on a labeled source-domain dataset to an unlabeled target-domain dataset.
We propose an end-to-end structured domain adaptation framework with an online relation-consistency regularization term.
Our proposed framework is shown to achieve state-of-the-art performance on multiple UDA tasks of person re-ID.
arXiv Detail & Related papers (2020-03-14T14:45:18Z) - CrDoCo: Pixel-level Domain Transfer with Cross-Domain Consistency [119.45667331836583]
Unsupervised domain adaptation algorithms aim to transfer the knowledge learned from one domain to another.
We present a novel pixel-wise adversarial domain adaptation algorithm.
arXiv Detail & Related papers (2020-01-09T19:00:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.