Controllable Style Transfer via Test-time Training of Implicit Neural Representation

• URL: http://arxiv.org/abs/2210.07762v2
• Date: Mon, 17 Oct 2022 06:30:48 GMT
• Title: Controllable Style Transfer via Test-time Training of Implicit Neural Representation
• Authors: Sunwoo Kim and Youngjo Min and Younghun Jung and Seungryong Kim
• Abstract summary: We propose a controllable style transfer framework based on Implicit Neural Representation that pixel-wisely controls the stylized output via test-time training. After being test-time trained once, thanks to the flexibility of the INR-based model, our framework can precisely control the stylized images in a pixel-wise manner and freely adjust image resolution without further optimization or training.
• Score: 34.880651923701066
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We propose a controllable style transfer framework based on Implicit Neural Representation that pixel-wisely controls the stylized output via test-time training. Unlike traditional image optimization methods that often suffer from unstable convergence and learning-based methods that require intensive training and have limited generalization ability, we present a model optimization framework that optimizes the neural networks during test-time with explicit loss functions for style transfer. After being test-time trained once, thanks to the flexibility of the INR-based model, our framework can precisely control the stylized images in a pixel-wise manner and freely adjust image resolution without further optimization or training. We demonstrate several applications.

Related papers

• Harnessing the Latent Diffusion Model for Training-Free Image Style Transfer [24.46409405016844]
  Style transfer task is one of those challenges that transfers the visual attributes of a style image to another content image. We propose a training-free style transfer algorithm, Style Tracking Reverse Diffusion Process (STRDP) for a pretrained Latent Diffusion Model (LDM) Our algorithm employs Adaptive Instance Normalization (AdaIN) function in a distinct manner during the reverse diffusion process of an LDM.
  arXiv  Detail & Related papers  (2024-10-02T09:28:21Z)
• Preconditioners for the Stochastic Training of Implicit Neural Representations [30.92757082348805]
  Implicit neural representations have emerged as a powerful technique for encoding complex continuous multidimensional signals as neural networks. We propose training using diagonal preconditioners, showcasing their effectiveness across various signal modalities.
  arXiv  Detail & Related papers  (2024-02-13T20:46:37Z)
• A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning [84.8813842101747]
  Unified Contrastive Arbitrary Style Transfer (UCAST) is a novel style representation learning and transfer framework. We present an adaptive contrastive learning scheme for style transfer by introducing an input-dependent temperature. Our framework consists of three key components, i.e., a parallel contrastive learning scheme for style representation and style transfer, a domain enhancement module for effective learning of style distribution, and a generative network for style transfer.
  arXiv  Detail & Related papers  (2023-03-09T04:35:00Z)
• Neural Data-Dependent Transform for Learned Image Compression [72.86505042102155]
  We build a neural data-dependent transform and introduce a continuous online mode decision mechanism to jointly optimize the coding efficiency for each individual image. The experimental results show the effectiveness of the proposed neural-syntax design and the continuous online mode decision mechanism.
  arXiv  Detail & Related papers  (2022-03-09T14:56:48Z)
• Deep Translation Prior: Test-time Training for Photorealistic Style Transfer [36.82737412912885]
  Recent techniques to solve photorealistic style transfer within deep convolutional neural networks (CNNs) generally require intensive training from large-scale datasets. We propose a novel framework, dubbed Deep Translation Prior (DTP), to accomplish photorealistic style transfer through test-time training on given input image pair with untrained networks.
  arXiv  Detail & Related papers  (2021-12-12T04:54:27Z)
• Image-specific Convolutional Kernel Modulation for Single Image Super-resolution [85.09413241502209]
  In this issue, we propose a novel image-specific convolutional modulation kernel (IKM) We exploit the global contextual information of image or feature to generate an attention weight for adaptively modulating the convolutional kernels. Experiments on single image super-resolution show that the proposed methods achieve superior performances over state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-16T11:05:10Z)
• Online Exemplar Fine-Tuning for Image-to-Image Translation [32.556050882376965]
  Existing techniques to solve exemplar-based image-to-image translation within deep convolutional neural networks (CNNs) generally require a training phase to optimize the network parameters. We propose a novel framework, for the first time, to solve exemplar-based translation through an online optimization given an input image pair. Our framework does not require the off-line training phase, which has been the main challenge of existing methods, but the pre-trained networks to enable optimization in online.
  arXiv  Detail & Related papers  (2020-11-18T15:13:16Z)
• Learning to Learn Parameterized Classification Networks for Scalable Input Images [76.44375136492827]
  Convolutional Neural Networks (CNNs) do not have a predictable recognition behavior with respect to the input resolution change. We employ meta learners to generate convolutional weights of main networks for various input scales. We further utilize knowledge distillation on the fly over model predictions based on different input resolutions.
  arXiv  Detail & Related papers  (2020-07-13T04:27:25Z)
• Learning Deformable Image Registration from Optimization: Perspective, Modules, Bilevel Training and Beyond [62.730497582218284]
  We develop a new deep learning based framework to optimize a diffeomorphic model via multi-scale propagation. We conduct two groups of image registration experiments on 3D volume datasets including image-to-atlas registration on brain MRI data and image-to-image registration on liver CT data.
  arXiv  Detail & Related papers  (2020-04-30T03:23:45Z)
• Self-supervised Fine-tuning for Correcting Super-Resolution Convolutional Neural Networks [17.922507191213494]
  We show that one can avoid training and correct for SR results with a fully self-supervised fine-tuning approach. We apply our fine-tuning algorithm on multiple image and video SR CNNs and show that it can successfully correct for a sub-optimal SR solution.
  arXiv  Detail & Related papers  (2019-12-30T11:02:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.