Improving Sketch Colorization using Adversarial Segmentation Consistency
- URL: http://arxiv.org/abs/2301.08590v1
- Date: Fri, 20 Jan 2023 14:07:30 GMT
- Title: Improving Sketch Colorization using Adversarial Segmentation Consistency
- Authors: Samet Hicsonmez, Nermin Samet, Emre Akbas, Pinar Duygulu
- Abstract summary: We propose a new method for producing color images from sketches.
We leverage semantic image segmentation from a general-purpose panoptic segmentation network to generate an additional adversarial loss function.
Our method is not restricted to datasets with segmentation labels and can be applied to unpaired translation tasks as well.
- Score: 12.55972766570669
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose a new method for producing color images from sketches. Current
solutions in sketch colorization either necessitate additional user instruction
or are restricted to the "paired" translation strategy. We leverage semantic
image segmentation from a general-purpose panoptic segmentation network to
generate an additional adversarial loss function. The proposed loss function is
compatible with any GAN model. Our method is not restricted to datasets with
segmentation labels and can be applied to unpaired translation tasks as well.
Using qualitative, and quantitative analysis, and based on a user study, we
demonstrate the efficacy of our method on four distinct image datasets. On the
FID metric, our model improves the baseline by up to 35 points. Our code,
pretrained models, scripts to produce newly introduced datasets and
corresponding sketch images are available at
https://github.com/giddyyupp/AdvSegLoss.
Related papers
- ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text [5.675944597452309]
We introduce two variations of an image-guided latent diffusion model utilizing different image tokens from the pre-trained CLIP image encoder.
We propose corresponding manipulation methods to adjust their results sequentially using weighted text inputs.
arXiv Detail & Related papers (2024-01-02T22:46:12Z) - Joint Dense-Point Representation for Contour-Aware Graph Segmentation [2.138299283227551]
We present a novel methodology that combines graph and dense segmentation techniques by jointly learning both point and pixel contour representations.
This addresses deficiencies in typical graph segmentation methods where misaligned objectives restrict the network from learning discriminative and contour features.
Our approach is validated on several Chest X-ray datasets, demonstrating clear improvements in segmentation stability and accuracy.
arXiv Detail & Related papers (2023-06-21T10:07:17Z) - ClipCrop: Conditioned Cropping Driven by Vision-Language Model [90.95403416150724]
We take advantage of vision-language models as a foundation for creating robust and user-intentional cropping algorithms.
We develop a method to perform cropping with a text or image query that reflects the user's intention as guidance.
Our pipeline design allows the model to learn text-conditioned aesthetic cropping with a small dataset.
arXiv Detail & Related papers (2022-11-21T14:27:07Z) - Learning to Annotate Part Segmentation with Gradient Matching [58.100715754135685]
This paper focuses on tackling semi-supervised part segmentation tasks by generating high-quality images with a pre-trained GAN.
In particular, we formulate the annotator learning as a learning-to-learn problem.
We show that our method can learn annotators from a broad range of labelled images including real images, generated images, and even analytically rendered images.
arXiv Detail & Related papers (2022-11-06T01:29:22Z) - From colouring-in to pointillism: revisiting semantic segmentation
supervision [48.637031591058175]
We propose a pointillist approach for semantic segmentation annotation, where only point-wise yes/no questions are answered.
We collected and released 22.6M point labels over 4,171 classes on the Open Images dataset.
arXiv Detail & Related papers (2022-10-25T16:42:03Z) - Estimating Appearance Models for Image Segmentation via Tensor
Factorization [0.0]
We propose a new approach to directly estimate appearance models from the image without prior information on the underlying segmentation.
Our method uses local high order color statistics from the image as an input to tensor factorization-based estimator for latent variable models.
This approach is able to estimate models in multiregion images and automatically output the regions proportions without prior user interaction.
arXiv Detail & Related papers (2022-08-16T17:21:00Z) - Distilling Ensemble of Explanations for Weakly-Supervised Pre-Training
of Image Segmentation Models [54.49581189337848]
We propose a method to enable the end-to-end pre-training for image segmentation models based on classification datasets.
The proposed method leverages a weighted segmentation learning procedure to pre-train the segmentation network en masse.
Experiment results show that, with ImageNet accompanied by PSSL as the source dataset, the proposed end-to-end pre-training strategy successfully boosts the performance of various segmentation models.
arXiv Detail & Related papers (2022-07-04T13:02:32Z) - Adversarial Segmentation Loss for Sketch Colorization [14.681690787310103]
We introduce a new method for generating color images from sketches or edge maps.
We argue that segmentation information could provide valuable guidance for sketch colorization.
Our model improves its baseline up to 35 points on the FID metric.
arXiv Detail & Related papers (2021-02-11T18:54:56Z) - Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation [49.90178055521207]
This work addresses weakly supervised semantic segmentation (WSSS), with the goal of bridging the gap between image-level annotations and pixel-level segmentation.
We formulate WSSS as a novel group-wise learning task that explicitly models semantic dependencies in a group of images to estimate more reliable pseudo ground-truths.
In particular, we devise a graph neural network (GNN) for group-wise semantic mining, wherein input images are represented as graph nodes.
arXiv Detail & Related papers (2020-12-09T12:40:13Z) - Sequential Graph Convolutional Network for Active Learning [53.99104862192055]
We propose a novel pool-based Active Learning framework constructed on a sequential Graph Convolution Network (GCN)
With a small number of randomly sampled images as seed labelled examples, we learn the parameters of the graph to distinguish labelled vs unlabelled nodes.
We exploit these characteristics of GCN to select the unlabelled examples which are sufficiently different from labelled ones.
arXiv Detail & Related papers (2020-06-18T00:55:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.