Transposer: Universal Texture Synthesis Using Feature Maps as Transposed Convolution Filter

• URL: http://arxiv.org/abs/2007.07243v1
• Date: Tue, 14 Jul 2020 17:57:59 GMT
• Title: Transposer: Universal Texture Synthesis Using Feature Maps as Transposed Convolution Filter
• Authors: Guilin Liu, Rohan Taori, Ting-Chun Wang, Zhiding Yu, Shiqiu Liu, Fitsum A. Reda, Karan Sapra, Andrew Tao, Bryan Catanzaro
• Abstract summary: We propose a novel way of using transposed convolution operation for texture synthesis. Our framework achieves state-of-the-art texture synthesis quality based on various metrics.
• Score: 43.9258342767253
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conventional CNNs for texture synthesis consist of a sequence of (de)-convolution and up/down-sampling layers, where each layer operates locally and lacks the ability to capture the long-term structural dependency required by texture synthesis. Thus, they often simply enlarge the input texture, rather than perform reasonable synthesis. As a compromise, many recent methods sacrifice generalizability by training and testing on the same single (or fixed set of) texture image(s), resulting in huge re-training time costs for unseen images. In this work, based on the discovery that the assembling/stitching operation in traditional texture synthesis is analogous to a transposed convolution operation, we propose a novel way of using transposed convolution operation. Specifically, we directly treat the whole encoded feature map of the input texture as transposed convolution filters and the features' self-similarity map, which captures the auto-correlation information, as input to the transposed convolution. Such a design allows our framework, once trained, to be generalizable to perform synthesis of unseen textures with a single forward pass in nearly real-time. Our method achieves state-of-the-art texture synthesis quality based on various metrics. While self-similarity helps preserve the input textures' regular structural patterns, our framework can also take random noise maps for irregular input textures instead of self-similarity maps as transposed convolution inputs. It allows to get more diverse results as well as generate arbitrarily large texture outputs by directly sampling large noise maps in a single pass as well.

Related papers

• Infinite Texture: Text-guided High Resolution Diffusion Texture Synthesis [61.189479577198846]
  We present Infinite Texture, a method for generating arbitrarily large texture images from a text prompt. Our approach fine-tunes a diffusion model on a single texture, and learns to embed that statistical distribution in the output domain of the model. At generation time, our fine-tuned diffusion model is used through a score aggregation strategy to generate output texture images of arbitrary resolution on a single GPU.
  arXiv  Detail & Related papers  (2024-05-13T21:53:09Z)
• Generating Non-Stationary Textures using Self-Rectification [70.91414475376698]
  This paper addresses the challenge of example-based non-stationary texture synthesis. We introduce a novel twostep approach wherein users first modify a reference texture using standard image editing tools. Our proposed method, termed "self-rectification", automatically refines this target into a coherent, seamless texture.
  arXiv  Detail & Related papers  (2024-01-05T15:07:05Z)
• Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering [47.78392889256976]
  Paint-it is a text-driven high-fidelity texture map synthesis method for 3D rendering. Paint-it synthesizes texture maps from a text description by synthesis-through-optimization, exploiting the Score-Distillation Sampling (SDS) We show that DC-PBR inherently schedules the optimization curriculum according to texture frequency and naturally filters out the noisy signals from SDS.
  arXiv  Detail & Related papers  (2023-12-18T17:17:08Z)
• Lightweight texture transfer based on texture feature preset [1.1863107884314108]
  We propose a lightweight texture transfer based on texture feature preset. The results show visually superior results but also reduces the model size by 3.2-3538 times and speeds up the process by 1.8-5.6 times.
  arXiv  Detail & Related papers  (2023-06-29T10:37:29Z)
• Paying U-Attention to Textures: Multi-Stage Hourglass Vision Transformer for Universal Texture Synthesis [2.8998926117101367]
  We present a novel U-Attention vision Transformer for universal texture synthesis. We exploit the natural long-range dependencies enabled by the attention mechanism to allow our approach to synthesize diverse textures. We propose a hierarchical hourglass backbone that attends to the global structure and performs patch mapping at varying scales.
  arXiv  Detail & Related papers  (2022-02-23T18:58:56Z)
• SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps [3.504542161036043]
  We present SeamlessGAN, a method capable of automatically generating tileable texture maps from a single input exemplar. In contrast to most existing methods, focused solely on solving the synthesis problem, our work tackles both problems, synthesis and tileability, simultaneously.
  arXiv  Detail & Related papers  (2022-01-13T18:24:26Z)
• FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting [77.8621673355983]
  We propose FuseFormer, a Transformer model designed for video inpainting via fine-grained feature fusion. We elaborately insert the soft composition and soft split into the feed-forward network, enabling the 1D linear layers to have the capability of modelling 2D structure. In both quantitative and qualitative evaluations, our proposed FuseFormer surpasses state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-09-07T10:13:29Z)
• A Generative Model for Texture Synthesis based on Optimal Transport between Feature Distributions [8.102785819558978]
  We show how to use our framework to learn a feed-forward neural network that can synthesize on-the-fly new textures of arbitrary size. We show how to use our framework to learn a feed-forward neural network that can synthesize on-the-fly new textures of arbitrary size in a very fast manner.
  arXiv  Detail & Related papers  (2020-06-19T13:32:55Z)
• Region-adaptive Texture Enhancement for Detailed Person Image Synthesis [86.69934638569815]
  RATE-Net is a novel framework for synthesizing person images with sharp texture details. The proposed framework leverages an additional texture enhancing module to extract appearance information from the source image. Experiments conducted on DeepFashion benchmark dataset have demonstrated the superiority of our framework compared with existing networks.
  arXiv  Detail & Related papers  (2020-05-26T02:33:21Z)
• Co-occurrence Based Texture Synthesis [25.4878061402506]
  We propose a fully convolutional generative adversarial network, conditioned locally on co-occurrence statistics, to generate arbitrarily large images. We show that our solution offers a stable, intuitive and interpretable latent representation for texture synthesis.
  arXiv  Detail & Related papers  (2020-05-17T08:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.