Towards Ultra-Resolution Neural Style Transfer via Thumbnail Instance Normalization

• URL: http://arxiv.org/abs/2103.11784v1
• Date: Mon, 22 Mar 2021 12:54:01 GMT
• Title: Towards Ultra-Resolution Neural Style Transfer via Thumbnail Instance Normalization
• Authors: Zhe Chen, Wenhai Wang, Enze Xie, Tong Lu, Ping Luo
• Abstract summary: We present an extremely simple Ultra-Resolution Style Transfer framework, termed URST, to flexibly process arbitrary high-resolution images. Most of the existing state-of-the-art methods would fall short due to massive memory cost and small stroke size when processing ultra-high resolution images.
• Score: 42.84367334160332
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present an extremely simple Ultra-Resolution Style Transfer framework, termed URST, to flexibly process arbitrary high-resolution images (e.g., 10000x10000 pixels) style transfer for the first time. Most of the existing state-of-the-art methods would fall short due to massive memory cost and small stroke size when processing ultra-high resolution images. URST completely avoids the memory problem caused by ultra-high resolution images by 1) dividing the image into small patches and 2) performing patch-wise style transfer with a novel Thumbnail Instance Normalization (TIN). Specifically, TIN can extract thumbnail's normalization statistics and apply them to small patches, ensuring the style consistency among different patches. Overall, the URST framework has three merits compared to prior arts. 1) We divide input image into small patches and adopt TIN, successfully transferring image style with arbitrary high-resolution. 2) Experiments show that our URST surpasses existing SOTA methods on ultra-high resolution images benefiting from the effectiveness of the proposed stroke perceptual loss in enlarging the stroke size. 3) Our URST can be easily plugged into most existing style transfer methods and directly improve their performance even without training. Code is available at https://github.com/czczup/URST.

Related papers

• MoreStyle: Relax Low-frequency Constraint of Fourier-based Image Reconstruction in Generalizable Medical Image Segmentation [53.24011398381715]
  We introduce a Plug-and-Play module for data augmentation called MoreStyle. MoreStyle diversifies image styles by relaxing low-frequency constraints in Fourier space. With the help of adversarial learning, MoreStyle pinpoints the most intricate style combinations within latent features.
  arXiv  Detail & Related papers  (2024-03-18T11:38:47Z)
• Scaling Painting Style Transfer [10.059627473725508]
  Neural style transfer (NST) is a technique that produces an unprecedentedly rich style transfer from a style image to a content image. This paper presents a solution to solve the original global optimization for ultra-high resolution (UHR) images. We show that our method produces style transfer of unmatched quality for such high-resolution painting styles.
  arXiv  Detail & Related papers  (2022-12-27T12:03:38Z)
• Iterative Patch Selection for High-Resolution Image Recognition [10.847032625429717]
  We propose a simple method, Iterative Patch Selection (IPS), which decouples the memory usage from the input size. IPS achieves this by selecting only the most salient patches, which are then aggregated into a global representation for image recognition. Our method demonstrates strong performance and has wide applicability across different domains, training regimes and image sizes while using minimal accelerator memory.
  arXiv  Detail & Related papers  (2022-10-24T07:55:57Z)
• HIPA: Hierarchical Patch Transformer for Single Image Super Resolution [62.7081074931892]
  This paper presents HIPA, a novel Transformer architecture that progressively recovers the high resolution image using a hierarchical patch partition. We build a cascaded model that processes an input image in multiple stages, where we start with tokens with small patch sizes and gradually merge to the full resolution. Such a hierarchical patch mechanism not only explicitly enables feature aggregation at multiple resolutions but also adaptively learns patch-aware features for different image regions.
  arXiv  Detail & Related papers  (2022-03-19T05:09:34Z)
• Implicit Neural Representations for Image Compression [103.78615661013623]
  Implicit Neural Representations (INRs) have gained attention as a novel and effective representation for various data types. We propose the first comprehensive compression pipeline based on INRs including quantization, quantization-aware retraining and entropy coding. We find that our approach to source compression with INRs vastly outperforms similar prior work.
  arXiv  Detail & Related papers  (2021-12-08T13:02:53Z)
• Beyond Fine-tuning: Classifying High Resolution Mammograms using Function-Preserving Transformations [32.40975574666405]
  The task of classifying mammograms is very challenging because the lesion is usually small in the high resolution image. In this paper, we propose to go beyond fine-tuning by introducing a novel framework called MorphHR. The proposed framework is to integrate function-preserving transformations, for any continuous non-linear activation neurons, to internally regularise the network for improving mammograms classification.
  arXiv  Detail & Related papers  (2021-01-20T03:04:07Z)
• Block Shuffle: A Method for High-resolution Fast Style Transfer with Limited Memory [4.511923587827301]
  Fast Style Transfer is a series of Neural Style Transfer algorithms that use feed-forward neural networks to render input images. Because of the high dimension of the output layer, these networks require much memory for computation. We propose a novel image synthesis method named emphblock shuffle, which converts a single task with high memory consumption to multiple subtasks with low memory consumption.
  arXiv  Detail & Related papers  (2020-08-09T10:33:21Z)
• SRFlow: Learning the Super-Resolution Space with Normalizing Flow [176.07982398988747]
  Super-resolution is an ill-posed problem, since it allows for multiple predictions for a given low-resolution image. We propose SRFlow: a normalizing flow based super-resolution method capable of learning the conditional distribution of the output. Our model is trained in a principled manner using a single loss, namely the negative log-likelihood.
  arXiv  Detail & Related papers  (2020-06-25T06:34:04Z)
• Collaborative Distillation for Ultra-Resolution Universal Style Transfer [71.18194557949634]
  We present a new knowledge distillation method (named Collaborative Distillation) for encoder-decoder based neural style transfer. We achieve ultra-resolution (over 40 megapixels) universal style transfer on a 12GB GPU for the first time.
  arXiv  Detail & Related papers  (2020-03-18T18:59:31Z)
• Remove Appearance Shift for Ultrasound Image Segmentation via Fast and Universal Style Transfer [13.355791568003559]
  We propose a novel and intuitive framework to remove the appearance shift, and hence improve the generalization ability of Deep Neural Networks (DNNs) We follow the spirit of universal style transfer to remove appearance shifts, which was not explored before for US images. Our framework achieved real-time speed required in the clinical US scanning.
  arXiv  Detail & Related papers  (2020-02-14T02:00:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.