BAGS: Building Animatable Gaussian Splatting from a Monocular Video with Diffusion Priors

• URL: http://arxiv.org/abs/2403.11427v1
• Date: Mon, 18 Mar 2024 02:44:46 GMT
• Title: BAGS: Building Animatable Gaussian Splatting from a Monocular Video with Diffusion Priors
• Authors: Tingyang Zhang, Qingzhe Gao, Weiyu Li, Libin Liu, Baoquan Chen,
• Abstract summary: We propose a method to build animatable 3D Gaussian Splatting from monocular video with diffusion priors. The 3D Gaussian representations significantly accelerate the training and rendering process, and the diffusion priors allow the method to learn 3D models with limited viewpoints. We perform an extensive evaluation across various real-world videos, demonstrating its superior performance compared to the current state-of-the-art methods.
• Score: 28.033944084753035
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Animatable 3D reconstruction has significant applications across various fields, primarily relying on artists' handcraft creation. Recently, some studies have successfully constructed animatable 3D models from monocular videos. However, these approaches require sufficient view coverage of the object within the input video and typically necessitate significant time and computational costs for training and rendering. This limitation restricts the practical applications. In this work, we propose a method to build animatable 3D Gaussian Splatting from monocular video with diffusion priors. The 3D Gaussian representations significantly accelerate the training and rendering process, and the diffusion priors allow the method to learn 3D models with limited viewpoints. We also present the rigid regularization to enhance the utilization of the priors. We perform an extensive evaluation across various real-world videos, demonstrating its superior performance compared to the current state-of-the-art methods.

Related papers

• PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animation [49.91188543847175]
  We reformulate 3D animation as a field prediction task and introduce a text-driven framework that infers a time-varying 4D flow field acting on 3D Gaussians.<n>By leveraging large language models (LLMs) and vision-language models (VLMs) for function generation, our approach interprets arbitrary prompts and instantly updates color, opacity, and positions of 3D Gaussians in real time.
  arXiv  Detail & Related papers  (2025-06-01T17:22:59Z)
• IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular Videos [33.12653115668027]
  Our method generates Multiplane Images (MPIs) that ensure geometric consistency. Our approach directly generates the final output through a single denoising process. To effectively learn from monocular videos, we introduce a training mechanism that reconstructs the output MPI randomly in either the target or the reference camera space.
  arXiv  Detail & Related papers  (2025-04-27T08:56:02Z)
• Enhancing Monocular 3D Scene Completion with Diffusion Model [20.81599069390756]
  3D scene reconstruction is essential for applications in virtual reality, robotics, and autonomous driving. Traditional 3D Gaussian Splatting techniques rely on images captured from multiple viewpoints to achieve optimal performance. We introduce FlashDreamer, a novel approach for reconstructing a complete 3D scene from a single image.
  arXiv  Detail & Related papers  (2025-03-02T04:36:57Z)
• Wonderland: Navigating 3D Scenes from a Single Image [43.99037613068823]
  We introduce a large-scale reconstruction model that leverages latents from a video diffusion model to predict 3D Gaussian Splattings of scenes in a feed-forward manner. We train the 3D reconstruction model to operate on the video latent space with a progressive learning strategy, enabling the efficient generation of high-quality, wide-scope, and generic 3D scenes.
  arXiv  Detail & Related papers  (2024-12-16T18:58:17Z)
• You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale [42.67300636733286]
  We present See3D, a visual-conditional multi-view diffusion model trained on large-scale Internet videos for open-world 3D creation. The model aims to Get 3D knowledge by solely Seeing the visual contents from the vast and rapidly growing video data. Our numerical and visual comparisons on single and sparse reconstruction benchmarks show that See3D, trained on cost-effective and scalable video data, achieves notable zero-shot and open-world generation capabilities.
  arXiv  Detail & Related papers  (2024-12-09T17:44:56Z)
• Animate3D: Animating Any 3D Model with Multi-view Video Diffusion [47.05131487114018]
  Animate3D is a novel framework for animating any static 3D model. We introduce a framework combining reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the multi-view video diffusion priors for animating 3D objects.
  arXiv  Detail & Related papers  (2024-07-16T05:35:57Z)
• iHuman: Instant Animatable Digital Humans From Monocular Videos [16.98924995658091]
  We present a fast, simple, yet effective method for creating animatable 3D digital humans from monocular videos. This work achieves and illustrates the need of accurate 3D mesh-type modelling of the human body. Our method is faster by an order of magnitude (in terms of training time) than its closest competitor.
  arXiv  Detail & Related papers  (2024-07-15T18:51:51Z)
• MotionDreamer: Exploring Semantic Video Diffusion features for Zero-Shot 3D Mesh Animation [10.263762787854862]
  We propose a technique for automatic re-animation of various 3D shapes based on a motion prior extracted from a video diffusion model. We leverage an explicit mesh-based representation compatible with existing computer-graphics pipelines. Our time-efficient zero-shot method achieves a superior performance re-animating a diverse set of 3D shapes.
  arXiv  Detail & Related papers  (2024-05-30T15:30:38Z)
• Bootstrap 3D Reconstructed Scenes from 3D Gaussian Splatting [10.06208115191838]
  We present a bootstrapping method to enhance the rendering of novel views using trained 3D-GS. Our results indicate that bootstrapping effectively reduces artifacts, as well as clear enhancements on the evaluation metrics.
  arXiv  Detail & Related papers  (2024-04-29T12:57:05Z)
• SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior [53.52396082006044]
  Current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. We propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model.
  arXiv  Detail & Related papers  (2024-03-29T09:20:29Z)
• V3D: Video Diffusion Models are Effective 3D Generators [19.33837029942662]
  We introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. Our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views.
  arXiv  Detail & Related papers  (2024-03-11T14:03:36Z)
• Animate124: Animating One Image to 4D Dynamic Scene [108.17635645216214]
  Animate124 is the first work to animate a single in-the-wild image into 3D video through textual motion descriptions. Our method demonstrates significant advancements over existing baselines.
  arXiv  Detail & Related papers  (2023-11-24T16:47:05Z)
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm [114.47216525866435]
  We introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation. For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness.
  arXiv  Detail & Related papers  (2023-10-12T17:59:57Z)
• Reconstructing Animatable Categories from Videos [65.14948977749269]
  Building animatable 3D models is challenging due to the need for 3D scans, laborious registration, and manual rigging. We present RAC that builds category 3D models from monocular videos while disentangling variations over instances and motion over time. We show that 3D models of humans, cats, and dogs can be learned from 50-100 internet videos.
  arXiv  Detail & Related papers  (2023-05-10T17:56:21Z)
• Learning 3D Photography Videos via Self-supervised Diffusion on Single Images [105.81348348510551]
  3D photography renders a static image into a video with appealing 3D visual effects. Existing approaches typically first conduct monocular depth estimation, then render the input frame to subsequent frames with various viewpoints. We present a novel task: out-animation, which extends the space and time of input objects.
  arXiv  Detail & Related papers  (2023-02-21T16:18:40Z)
• 3D-Aware Video Generation [149.5230191060692]
  We explore 4D generative adversarial networks (GANs) that learn generation of 3D-aware videos. By combining neural implicit representations with time-aware discriminator, we develop a GAN framework that synthesizes 3D video supervised only with monocular videos.
  arXiv  Detail & Related papers  (2022-06-29T17:56:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.