4DStyleGaussian: Zero-shot 4D Style Transfer with Gaussian Splatting

• URL: http://arxiv.org/abs/2410.10412v1
• Date: Mon, 14 Oct 2024 12:03:00 GMT
• Title: 4DStyleGaussian: Zero-shot 4D Style Transfer with Gaussian Splatting
• Authors: Wanlin Liang, Hongbin Xu, Weitao Chen, Feng Xiao, Wenxiong Kang,
• Abstract summary: We introduce 4DStyleGaussian, a novel 4D style transfer framework to achieve real-time stylization of arbitrary style references. Our method can achieve high-quality and zero-shot stylization for 4D scenarios with enhanced efficiency and spatial-temporal consistency.
• Score: 15.456479631131522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D neural style transfer has gained significant attention for its potential to provide user-friendly stylization with spatial consistency. However, existing 3D style transfer methods often fall short in terms of inference efficiency, generalization ability, and struggle to handle dynamic scenes with temporal consistency. In this paper, we introduce 4DStyleGaussian, a novel 4D style transfer framework designed to achieve real-time stylization of arbitrary style references while maintaining reasonable content affinity, multi-view consistency, and temporal coherence. Our approach leverages an embedded 4D Gaussian Splatting technique, which is trained using a reversible neural network for reducing content loss in the feature distillation process. Utilizing the 4D embedded Gaussians, we predict a 4D style transformation matrix that facilitates spatially and temporally consistent style transfer with Gaussian Splatting. Experiments demonstrate that our method can achieve high-quality and zero-shot stylization for 4D scenarios with enhanced efficiency and spatial-temporal consistency.

Related papers

• S4D: Streaming 4D Real-World Reconstruction with Gaussians and 3D Control Points [30.46796069720543]
  We introduce a novel approach for streaming 4D real-world reconstruction utilizing discrete 3D control points. This method physically models local rays and establishes a motion-decoupling coordinate system. By effectively merging traditional graphics with learnable pipelines, it provides a robust and efficient local 6-degrees-of-freedom (6 DoF) motion representation.
  arXiv  Detail & Related papers  (2024-08-23T12:51:49Z)
• Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models [116.31344506738816]
  We present a novel framework, textbfDiffusion4D, for efficient and scalable 4D content generation. We develop a 4D-aware video diffusion model capable of synthesizing orbital views of dynamic 3D assets. Our method surpasses prior state-of-the-art techniques in terms of generation efficiency and 4D geometry consistency.
  arXiv  Detail & Related papers  (2024-05-26T17:47:34Z)
• SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer [57.506654943449796]
  We propose an efficient, sparse-controlled video-to-4D framework named SC4D that decouples motion and appearance. Our method surpasses existing methods in both quality and efficiency. We devise a novel application that seamlessly transfers motion onto a diverse array of 4D entities.
  arXiv  Detail & Related papers  (2024-04-04T18:05:18Z)
• STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians [36.83603109001298]
  STAG4D is a novel framework that combines pre-trained diffusion models with dynamic 3D Gaussian splatting for high-fidelity 4D generation. We show that our method outperforms prior 4D generation works in rendering quality, spatial-temporal consistency, and generation robustness.
  arXiv  Detail & Related papers  (2024-03-22T04:16:33Z)
• GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation [28.780488884938997]
  We introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting. Our method achieves state-of-the-art results on both tasks of 4D generation and 4D novel view synthesis.
  arXiv  Detail & Related papers  (2024-03-19T02:22:21Z)
• StyleDyRF: Zero-shot 4D Style Transfer for Dynamic Neural Radiance Fields [21.55426133036809]
  Existing efforts on 3D style transfer can effectively combine the visual features of style images and neural radiance fields (NeRF) We introduce StyleDyRF, a method that represents the 4D feature space by deforming a canonical feature volume. We show that our method not only renders 4D photorealistic style transfer results in a zero-shot manner but also outperforms existing methods in terms of visual quality and consistency.
  arXiv  Detail & Related papers  (2024-03-13T07:42:21Z)
• StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting [141.05924680451804]
  StyleGaussian is a novel 3D style transfer technique. It allows instant transfer of any image's style to a 3D scene at 10 frames per second (fps)
  arXiv  Detail & Related papers  (2024-03-12T16:44:52Z)
• 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency [118.15258850780417]
  This work introduces 4DGen, a novel framework for grounded 4D content creation. We identify static 3D assets and monocular video sequences as key components in constructing the 4D content. Our pipeline facilitates conditional 4D generation, enabling users to specify geometry (3D assets) and motion (monocular videos)
  arXiv  Detail & Related papers  (2023-12-28T18:53:39Z)
• DreamGaussian4D: Generative 4D Gaussian Splatting [56.49043443452339]
  We introduce DreamGaussian4D (DG4D), an efficient 4D generation framework that builds on Gaussian Splatting (GS) Our key insight is that combining explicit modeling of spatial transformations with static GS makes an efficient and powerful representation for 4D generation. Video generation methods have the potential to offer valuable spatial-temporal priors, enhancing the high-quality 4D generation.
  arXiv  Detail & Related papers  (2023-12-28T17:16:44Z)
• Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models [94.07744207257653]
  We focus on the underexplored text-to-4D setting and synthesize dynamic, animated 3D objects. We combine text-to-image, text-to-video, and 3D-aware multiview diffusion models to provide feedback during 4D object optimization.
  arXiv  Detail & Related papers  (2023-12-21T11:41:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.