Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenes

• URL: http://arxiv.org/abs/2411.19233v2
• Date: Fri, 07 Mar 2025 12:37:47 GMT
• Title: Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenes
• Authors: Thomas Wimmer, Michael Oechsle, Michael Niemeyer, Federico Tombari,
• Abstract summary: We propose a method for animating parts of high-quality 3D scenes in a Gaussian Splatting representation.<n>We find that, in contrast to prior work, this enables realistic animations of complex, pre-existing 3D scenes.
• Score: 49.26872036160368
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State-of-the-art novel view synthesis methods achieve impressive results for multi-view captures of static 3D scenes. However, the reconstructed scenes still lack "liveliness," a key component for creating engaging 3D experiences. Recently, novel video diffusion models generate realistic videos with complex motion and enable animations of 2D images, however they cannot naively be used to animate 3D scenes as they lack multi-view consistency. To breathe life into the static world, we propose Gaussians2Life, a method for animating parts of high-quality 3D scenes in a Gaussian Splatting representation. Our key idea is to leverage powerful video diffusion models as the generative component of our model and to combine these with a robust technique to lift 2D videos into meaningful 3D motion. We find that, in contrast to prior work, this enables realistic animations of complex, pre-existing 3D scenes and further enables the animation of a large variety of object classes, while related work is mostly focused on prior-based character animation, or single 3D objects. Our model enables the creation of consistent, immersive 3D experiences for arbitrary scenes.

Related papers

• Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation [22.325990468075368]
  Animators use the 2D sketches in storyboards as references to craft the desired 3D animations through a trial-and-error process. There is a high demand for automated methods that can directly translate 2D storyboard sketches into 3D animations. We present Sketch2Anim, composed of two key modules for sketch constraint understanding and motion generation.
  arXiv  Detail & Related papers  (2025-04-27T10:38:17Z)
• Animating the Uncaptured: Humanoid Mesh Animation with Video Diffusion Models [71.78723353724493]
  Animation of humanoid characters is essential in various graphics applications. We propose an approach to synthesize 4D animated sequences of input static 3D humanoid meshes.
  arXiv  Detail & Related papers  (2025-03-20T10:00:22Z)
• MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling [21.1274747033854]
  Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. Milo is a novel framework which can synthesize character videos with controllable attributes. Milo achieves advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes.
  arXiv  Detail & Related papers  (2024-09-24T15:00:07Z)
• Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches [50.51643519253066]
  3D Content Generation is at the heart of many computer graphics applications, including video gaming, film-making, virtual and augmented reality, etc. This paper proposes a novel deep-learning based approach for automatically generating interactive and playable 3D game scenes.
  arXiv  Detail & Related papers  (2024-08-08T16:27:37Z)
• LoopGaussian: Creating 3D Cinemagraph with Multi-view Images via Eulerian Motion Field [13.815932949774858]
  Cinemagraph is a form of visual media that combines elements of still photography and subtle motion to create a captivating experience. We propose LoopGaussian to elevate cinemagraph from 2D image space to 3D space using 3D Gaussian modeling. Experiment results validate the effectiveness of our approach, demonstrating high-quality and visually appealing scene generation.
  arXiv  Detail & Related papers  (2024-04-13T11:07:53Z)
• Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis [88.17520303867099]
  One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from an unseen image, and then animate it with a reference video or audio. We present Real3D-Potrait, a framework that improves the one-shot 3D reconstruction power with a large image-to-plane model. Experiments show that Real3D-Portrait generalizes well to unseen identities and generates more realistic talking portrait videos.
  arXiv  Detail & Related papers  (2024-01-16T17:04:30Z)
• CC3D: Layout-Conditioned Generation of Compositional 3D Scenes [49.281006972028194]
  We introduce CC3D, a conditional generative model that synthesizes complex 3D scenes conditioned on 2D semantic scene layouts. Our evaluations on synthetic 3D-FRONT and real-world KITTI-360 datasets demonstrate that our model generates scenes of improved visual and geometric quality.
  arXiv  Detail & Related papers  (2023-03-21T17:59:02Z)
• 3D Cinemagraphy from a Single Image [73.09720823592092]
  We present 3D Cinemagraphy, a new technique that marries 2D image animation with 3D photography. Given a single still image as input, our goal is to generate a video that contains both visual content animation and camera motion.
  arXiv  Detail & Related papers  (2023-03-10T06:08:23Z)
• Unsupervised Volumetric Animation [54.52012366520807]
  We propose a novel approach for unsupervised 3D animation of non-rigid deformable objects. Our method learns the 3D structure and dynamics of objects solely from single-view RGB videos. We show our model can obtain animatable 3D objects from a single volume or few images.
  arXiv  Detail & Related papers  (2023-01-26T18:58:54Z)
• Physically Plausible Animation of Human Upper Body from a Single Image [41.027391105867345]
  We present a new method for generating controllable, dynamically responsive, and photorealistic human animations. Given an image of a person, our system allows the user to generate Physically plausible Upper Body Animation (PUBA) using interaction in the image space.
  arXiv  Detail & Related papers  (2022-12-09T09:36:59Z)
• Action2video: Generating Videos of Human 3D Actions [31.665831044217363]
  We aim to tackle the interesting yet challenging problem of generating videos of diverse and natural human motions from prescribed action categories. Key issue lies in the ability to synthesize multiple distinct motion sequences that are realistic in their visual appearances. Action2motionally generates plausible 3D pose sequences of a prescribed action category, which are processed and rendered by motion2video to form 2D videos.
  arXiv  Detail & Related papers  (2021-11-12T20:20:37Z)
• Unsupervised object-centric video generation and decomposition in 3D [36.08064849807464]
  We propose to model a video as the view seen while moving through a scene with multiple 3D objects and a 3D background. Our model is trained from monocular videos without any supervision, yet learns to generate coherent 3D scenes containing several moving objects.
  arXiv  Detail & Related papers  (2020-07-07T18:01:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.