Dynamic Texture Synthesis by Incorporating Long-range Spatial and Temporal Correlations

• URL: http://arxiv.org/abs/2104.05940v2
• Date: Wed, 14 Apr 2021 04:15:31 GMT
• Title: Dynamic Texture Synthesis by Incorporating Long-range Spatial and Temporal Correlations
• Authors: Kaitai Zhang, Bin Wang, Hong-Shuo Chen, Ye Wang, Shiyu Mou, and C.-C. Jay Kuo
• Abstract summary: We introduce a new loss term, called the Shifted Gram loss, to capture the structural and long-range correlation of the reference texture video. We also introduce a frame sampling strategy to exploit long-period motion across multiple frames.
• Score: 27.247382497265214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The main challenge of dynamic texture synthesis lies in how to maintain spatial and temporal consistency in synthesized videos. The major drawback of existing dynamic texture synthesis models comes from poor treatment of the long-range texture correlation and motion information. To address this problem, we incorporate a new loss term, called the Shifted Gram loss, to capture the structural and long-range correlation of the reference texture video. Furthermore, we introduce a frame sampling strategy to exploit long-period motion across multiple frames. With these two new techniques, the application scope of existing texture synthesis models can be extended. That is, they can synthesize not only homogeneous but also structured dynamic texture patterns. Thorough experimental results are provided to demonstrate that our proposed dynamic texture synthesis model offers state-of-the-art visual performance.

Related papers

• Diffusion Priors for Dynamic View Synthesis from Monocular Videos [59.42406064983643]
  Dynamic novel view synthesis aims to capture the temporal evolution of visual content within videos. We first finetune a pretrained RGB-D diffusion model on the video frames using a customization technique. We distill the knowledge from the finetuned model to a 4D representations encompassing both dynamic and static Neural Radiance Fields.
  arXiv  Detail & Related papers  (2024-01-10T23:26:41Z)
• Motion-Conditioned Diffusion Model for Controllable Video Synthesis [75.367816656045]
  We introduce MCDiff, a conditional diffusion model that generates a video from a starting image frame and a set of strokes. We show that MCDiff achieves the state-the-art visual quality in stroke-guided controllable video synthesis.
  arXiv  Detail & Related papers  (2023-04-27T17:59:32Z)
• ContraNeRF: Generalizable Neural Radiance Fields for Synthetic-to-real Novel View Synthesis via Contrastive Learning [102.46382882098847]
  We first investigate the effects of synthetic data in synthetic-to-real novel view synthesis. We propose to introduce geometry-aware contrastive learning to learn multi-view consistent features with geometric constraints. Our method can render images with higher quality and better fine-grained details, outperforming existing generalizable novel view synthesis methods in terms of PSNR, SSIM, and LPIPS.
  arXiv  Detail & Related papers  (2023-03-20T12:06:14Z)
• Leaping Into Memories: Space-Time Deep Feature Synthesis [93.10032043225362]
  We propose LEAPS, an architecture-independent method for synthesizing videos from internal models. We quantitatively and qualitatively evaluate the applicability of LEAPS by inverting a range of architectures convolutional attention-based on Kinetics-400.
  arXiv  Detail & Related papers  (2023-03-17T12:55:22Z)
• A geometrically aware auto-encoder for multi-texture synthesis [1.2891210250935146]
  We propose an auto-encoder architecture for multi-texture synthesis. Images are embedded in a compact and geometrically consistent latent space. Texture synthesis and tasks can be performed directly from these latent codes.
  arXiv  Detail & Related papers  (2023-02-03T09:28:39Z)
• Texture Representation via Analysis and Synthesis with Generative Adversarial Networks [11.67779950826776]
  We investigate data-driven texture modeling via analysis and synthesis with generative synthesis. We adopt StyleGAN3 for synthesis and demonstrate that it produces diverse textures beyond those represented in the training data. For texture analysis, we propose GAN using a novel latent consistency criterion for synthesized textures, and iterative refinement with Gramian loss for real textures.
  arXiv  Detail & Related papers  (2022-12-20T03:57:11Z)
• Paying U-Attention to Textures: Multi-Stage Hourglass Vision Transformer for Universal Texture Synthesis [2.8998926117101367]
  We present a novel U-Attention vision Transformer for universal texture synthesis. We exploit the natural long-range dependencies enabled by the attention mechanism to allow our approach to synthesize diverse textures. We propose a hierarchical hourglass backbone that attends to the global structure and performs patch mapping at varying scales.
  arXiv  Detail & Related papers  (2022-02-23T18:58:56Z)
• Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes [70.76742458931935]
  We introduce a new representation that models the dynamic scene as a time-variant continuous function of appearance, geometry, and 3D scene motion. Our representation is optimized through a neural network to fit the observed input views. We show that our representation can be used for complex dynamic scenes, including thin structures, view-dependent effects, and natural degrees of motion.
  arXiv  Detail & Related papers  (2020-11-26T01:23:44Z)
• Co-occurrence Based Texture Synthesis [25.4878061402506]
  We propose a fully convolutional generative adversarial network, conditioned locally on co-occurrence statistics, to generate arbitrarily large images. We show that our solution offers a stable, intuitive and interpretable latent representation for texture synthesis.
  arXiv  Detail & Related papers  (2020-05-17T08:01:44Z)
• Hierarchy Composition GAN for High-fidelity Image Synthesis [57.32311953820988]
  This paper presents an innovative Hierarchical Composition GAN (HIC-GAN) HIC-GAN incorporates image synthesis in geometry and appearance domains into an end-to-end trainable network. Experiments on scene text image synthesis, portrait editing and indoor rendering tasks show that the proposed HIC-GAN achieves superior synthesis performance qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2019-05-12T11:11:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.