ZDySS -- Zero-Shot Dynamic Scene Stylization using Gaussian Splatting

• URL: http://arxiv.org/abs/2501.03875v1
• Date: Tue, 07 Jan 2025 15:39:02 GMT
• Title: ZDySS -- Zero-Shot Dynamic Scene Stylization using Gaussian Splatting
• Authors: Abhishek Saroha, Florian Hofherr, Mariia Gladkova, Cecilia Curreli, Or Litany, Daniel Cremers,
• Abstract summary: Stylizing a dynamic scene based on an exemplar image is critical for various real-world applications, including gaming filmmaking, and augmented and virtual reality.<n>We introduceDySS, a zero-shot stylization framework for dynamic scenes, allowing our model to generalize to previously unseen style images at inference.<n>Our method demonstrates performance and coherence over state-of-the-art baselines in tests on real-world dynamic scenes, making it a robust solution for practical applications.
• Score: 41.678269742147066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Stylizing a dynamic scene based on an exemplar image is critical for various real-world applications, including gaming, filmmaking, and augmented and virtual reality. However, achieving consistent stylization across both spatial and temporal dimensions remains a significant challenge. Most existing methods are designed for static scenes and often require an optimization process for each style image, limiting their adaptability. We introduce ZDySS, a zero-shot stylization framework for dynamic scenes, allowing our model to generalize to previously unseen style images at inference. Our approach employs Gaussian splatting for scene representation, linking each Gaussian to a learned feature vector that renders a feature map for any given view and timestamp. By applying style transfer on the learned feature vectors instead of the rendered feature map, we enhance spatio-temporal consistency across frames. Our method demonstrates superior performance and coherence over state-of-the-art baselines in tests on real-world dynamic scenes, making it a robust solution for practical applications.

Related papers

• StyleMe3D: Stylization with Disentangled Priors by Multiple Encoders on 3D Gaussians [23.1385740508835]
  StyleMe3D is a holistic framework for 3D GS style transfer. It integrates multi-modal style conditioning, multi-level semantic alignment, and perceptual quality enhancement. This work bridges photorealistic 3D GS and artistic stylization, unlocking applications in gaming, virtual worlds, and digital art.
  arXiv  Detail & Related papers  (2025-04-21T17:59:55Z)
• Divide-and-Conquer: Dual-Hierarchical Optimization for Semantic 4D Gaussian Spatting [16.15871890842964]
  We propose Dual-Hierarchical Optimization (DHO), which consists of Hierarchical Gaussian Flow and Hierarchical Gaussian Guidance. Our method consistently outperforms the baselines on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2025-03-25T03:46:13Z)
• Efficient Gaussian Splatting for Monocular Dynamic Scene Rendering via Sparse Time-Variant Attribute Modeling [64.84686527988809]
  Deformable Gaussian Splatting has emerged as a robust solution to represent real-world dynamic scenes. Our approach formulates dynamic scenes using a sparse anchor-grid representation, with the motion flow of dense Gaussians calculated via a classical kernel representation. Experiments on two real-world datasets demonstrate that our EDGS significantly improves the rendering speed with superior rendering quality.
  arXiv  Detail & Related papers  (2025-02-27T18:53:06Z)
• 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives [116.2042238179433]
  In this paper, we frame dynamic scenes as unconstrained 4D volume learning problems.<n>We represent a target dynamic scene using a collection of 4D Gaussian primitives with explicit geometry and appearance features.<n>This approach can capture relevant information in space and time by fitting the underlying photorealistic-temporal volume.<n> Notably, our 4DGS model is the first solution that supports real-time rendering of high-resolution, novel views for complex dynamic scenes.
  arXiv  Detail & Related papers  (2024-12-30T05:30:26Z)
• Gaussian Splatting in Style [32.41970914897462]
  3D sceneization extends the work of neural style transfer to 3D. A vital challenge in this problem is to maintain the uniformity of the stylized appearance across multiple views. We propose a novel architecture trained on a collection of style images that, at test time, produces real time high-quality stylized novel views.
  arXiv  Detail & Related papers  (2024-03-13T13:06:31Z)
• SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting [7.553079256251747]
  We extend 3D Gaussian Splatting to reconstruct dynamic scenes. We produce high-quality renderings of general dynamic scenes with competitive quantitative performance. Our method can be viewed in real-time in our dynamic interactive viewer.
  arXiv  Detail & Related papers  (2023-12-20T03:54:03Z)
• FlowIBR: Leveraging Pre-Training for Efficient Neural Image-Based Rendering of Dynamic Scenes [5.118560450410779]
  FlowIBR is a novel approach for efficient monocular novel view of dynamic scenes. It integrates a neural image-based rendering method, pre-trained on a large corpus of widely available static scenes, with a per-scene optimized scene flow field. The proposed method reduces per-scene optimization time by an order of magnitude, achieving comparable rendering quality to existing methods.
  arXiv  Detail & Related papers  (2023-09-11T12:35:17Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• Learning Diverse Tone Styles for Image Retouching [73.60013618215328]
  We propose to learn diverse image retouching with normalizing flow-based architectures. A joint-training pipeline is composed of a style encoder, a conditional RetouchNet, and the image tone style normalizing flow (TSFlow) module. Our proposed method performs favorably against state-of-the-art methods and is effective in generating diverse results.
  arXiv  Detail & Related papers  (2022-07-12T09:49:21Z)
• StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions [11.153966202832933]
  We apply style transfer on mesh reconstructions of indoor scenes. This enables VR applications like experiencing 3D environments painted in the style of a favorite artist.
  arXiv  Detail & Related papers  (2021-12-02T18:59:59Z)
• IBRNet: Learning Multi-View Image-Based Rendering [67.15887251196894]
  We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering.
  arXiv  Detail & Related papers  (2021-02-25T18:56:21Z)
• Real-time Localized Photorealistic Video Style Transfer [25.91181753178577]
  We present a novel algorithm for transferring artistic styles of semantically meaningful local regions of an image onto local regions of a target video. Our method, based on a deep neural network architecture inspired by recent work in photorealistic style transfer, is real-time and works on arbitrary inputs. We demonstrate our method on a variety of style images and target videos, including the ability to transfer different styles onto multiple objects simultaneously.
  arXiv  Detail & Related papers  (2020-10-20T06:21:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.