Towards High-resolution and Disentangled Reference-based Sketch Colorization

• URL: http://arxiv.org/abs/2603.05971v2
• Date: Mon, 09 Mar 2026 02:21:04 GMT
• Title: Towards High-resolution and Disentangled Reference-based Sketch Colorization
• Abstract summary: We present a framework that directly minimizes the distribution shift, thereby achieving superior quality, resolution, and controllability of colorization.<n>We adopt an anime-specific Tagger Network to extract fine-grained attributions from reference images and modulate SDXL's conditional encoders to ensure precise control.
• Score: 41.858933067874354
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sketch colorization is a critical task for automating and assisting in the creation of animations and digital illustrations. Previous research identified the primary difficulty as the distribution shift between semantically aligned training data and highly diverse test data, and focused on mitigating the artifacts caused by the distribution shift instead of fundamentally resolving the problem. In this paper, we present a framework that directly minimizes the distribution shift, thereby achieving superior quality, resolution, and controllability of colorization. We propose a dual-branch framework to explicitly model the data distributions of the training process and inference process with a semantic-aligned branch and a semantic-misaligned branch, respectively. A Gram Regularization Loss is applied across the feature maps of both branches, effectively enforcing cross-domain distribution coherence and stability. Furthermore, we adopt an anime-specific Tagger Network to extract fine-grained attributions from reference images and modulate SDXL's conditional encoders to ensure precise control, and a plugin module to enhance texture transfer. Quantitative and qualitative comparisons, alongside user studies, confirm that our method effectively overcomes the distribution shift challenge, establishing State-of-the-Art performance across both quality and controllability metrics. Ablation study reveals the influence of each component.

Related papers

• Redirecting the Flow: Image Customization through Attention Distribution Shift [18.601789249339014]
  Subject-driven image customization aims to generate images that follow textual instructions and preserve the identity of a given reference subject.<n>Existing approaches suffer from limited efficiency, misalignment between extracted reference features and the generative process, and interference from irrelevant information.<n>We propose CustomShift, a dual-branch architecture based on Stable Diffusion 3.<n> Experiments on the DreamBooth and Custom101 benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-06-15T15:44:14Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• A Theory-Inspired Framework for Few-Shot Cross-Modal Sketch Person Re-Identification [5.499165736807566]
  Sketch based person re-identification aims to match hand-drawn sketches with RGB surveillance images.<n>We introduce KTCAA, a framework for few-shot cross-modal generalization.<n>We show that KTCAA achieves state-of-the-art performance, particularly in data-scarce conditions.
  arXiv  Detail & Related papers  (2025-11-24T01:26:46Z)
• Knowledge Distillation and Enhanced Subdomain Adaptation Using Graph Convolutional Network for Resource-Constrained Bearing Fault Diagnosis [0.0]
  We propose a progressive knowledge distillation framework that transfers knowledge from a complex teacher model to a compact and efficient student model.<n>We introduce Enhanced Local Maximum Mean Squared Discrepancy (ELMMSD), which leverages mean and variance statistics in the Reproducing Kernel Hilbert Space (RKHS) and incorporates a priori probability distributions between labels.
  arXiv  Detail & Related papers  (2025-01-13T10:05:47Z)
• Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation [54.96563068182733]
  We propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task. MADM utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. We show that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities.
  arXiv  Detail & Related papers  (2024-10-29T03:49:40Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• A Simplified Framework for Contrastive Learning for Node Representations [2.277447144331876]
  We investigate the potential of deploying contrastive learning in combination with Graph Neural Networks for embedding nodes in a graph. We show that the quality of the resulting embeddings and training time can be significantly improved by a simple column-wise postprocessing of the embedding matrix. This modification yields improvements in downstream classification tasks of up to 1.5% and even beats existing state-of-the-art approaches on 6 out of 8 different benchmarks.
  arXiv  Detail & Related papers  (2023-05-01T02:04:36Z)
• Balancing Discriminability and Transferability for Source-Free Domain Adaptation [55.143687986324935]
  Conventional domain adaptation (DA) techniques aim to improve domain transferability by learning domain-invariant representations. The requirement of simultaneous access to labeled source and unlabeled target renders them unsuitable for the challenging source-free DA setting. We derive novel insights to show that a mixup between original and corresponding translated generic samples enhances the discriminability-transferability trade-off.
  arXiv  Detail & Related papers  (2022-06-16T09:06:22Z)
• Certifying Model Accuracy under Distribution Shifts [151.67113334248464]
  We present provable robustness guarantees on the accuracy of a model under bounded Wasserstein shifts of the data distribution. We show that a simple procedure that randomizes the input of the model within a transformation space is provably robust to distributional shifts under the transformation.
  arXiv  Detail & Related papers  (2022-01-28T22:03:50Z)
• Semi-Supervised Semantic Segmentation with Cross-Consistency Training [8.894935073145252]
  We present a novel cross-consistency based semi-supervised approach for semantic segmentation. Our method achieves state-of-the-art results in several datasets.
  arXiv  Detail & Related papers  (2020-03-19T20:10:37Z)
• When Relation Networks meet GANs: Relation GANs with Triplet Loss [110.7572918636599]
  Training stability is still a lingering concern of generative adversarial networks (GANs) In this paper, we explore a relation network architecture for the discriminator and design a triplet loss which performs better generalization and stability. Experiments on benchmark datasets show that the proposed relation discriminator and new loss can provide significant improvement on variable vision tasks.
  arXiv  Detail & Related papers  (2020-02-24T11:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.