Monocular Dynamic Gaussian Splatting is Fast and Brittle but Smooth Motion Helps

• URL: http://arxiv.org/abs/2412.04457v1
• Date: Thu, 05 Dec 2024 18:59:08 GMT
• Title: Monocular Dynamic Gaussian Splatting is Fast and Brittle but Smooth Motion Helps
• Authors: Yiqing Liang, Mikhail Okunev, Mikaela Angelina Uy, Runfeng Li, Leonidas Guibas, James Tompkin, Adam W. Harley,
• Abstract summary: We organize, benchmark, and analyze many Gaussian-splatting-based methods.<n>We quantify how their differences impact performance.<n>Fast rendering speed of all Gaussian-based methods comes at the cost of brittleness in optimization.
• Score: 14.35885714606969
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gaussian splatting methods are emerging as a popular approach for converting multi-view image data into scene representations that allow view synthesis. In particular, there is interest in enabling view synthesis for dynamic scenes using only monocular input data -- an ill-posed and challenging problem. The fast pace of work in this area has produced multiple simultaneous papers that claim to work best, which cannot all be true. In this work, we organize, benchmark, and analyze many Gaussian-splatting-based methods, providing apples-to-apples comparisons that prior works have lacked. We use multiple existing datasets and a new instructive synthetic dataset designed to isolate factors that affect reconstruction quality. We systematically categorize Gaussian splatting methods into specific motion representation types and quantify how their differences impact performance. Empirically, we find that their rank order is well-defined in synthetic data, but the complexity of real-world data currently overwhelms the differences. Furthermore, the fast rendering speed of all Gaussian-based methods comes at the cost of brittleness in optimization. We summarize our experiments into a list of findings that can help to further progress in this lively problem setting. Project Webpage: https://lynl7130.github.io/MonoDyGauBench.github.io/

Related papers

• HoliGS: Holistic Gaussian Splatting for Embodied View Synthesis [59.25751939710903]
  We propose a novel deformable Gaussian splatting framework that addresses embodied view synthesis from long monocular RGB videos.<n>Our method leverages invertible Gaussian Splatting deformation networks to reconstruct large-scale, dynamic environments accurately.<n>Results highlight a practical and scalable solution for EVS in real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-24T03:54:40Z)
• Divide-and-Conquer: Dual-Hierarchical Optimization for Semantic 4D Gaussian Spatting [16.15871890842964]
  We propose Dual-Hierarchical Optimization (DHO), which consists of Hierarchical Gaussian Flow and Hierarchical Gaussian Guidance. Our method consistently outperforms the baselines on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2025-03-25T03:46:13Z)
• Efficient Gaussian Splatting for Monocular Dynamic Scene Rendering via Sparse Time-Variant Attribute Modeling [64.84686527988809]
  Deformable Gaussian Splatting has emerged as a robust solution to represent real-world dynamic scenes. Our approach formulates dynamic scenes using a sparse anchor-grid representation, with the motion flow of dense Gaussians calculated via a classical kernel representation. Experiments on two real-world datasets demonstrate that our EDGS significantly improves the rendering speed with superior rendering quality.
  arXiv  Detail & Related papers  (2025-02-27T18:53:06Z)
• 4D Gaussian Splatting with Scale-aware Residual Field and Adaptive Optimization for Real-time Rendering of Temporally Complex Dynamic Scenes [19.24815625343669]
  SaRO-GS is a novel dynamic scene representation capable of achieving real-time rendering. To handle temporally complex dynamic scenes, we introduce a Scale-aware Residual Field. Our method has demonstrated state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-09T08:44:19Z)
• Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization [11.418632671254564]
  3D Gaussian Splatting has emerged as a powerful tool for fast and accurate novel-view synthesis from a set of posed input images. We propose an extension to the 3D Gaussian Splatting framework by optimizing the extrinsic camera parameters with respect to photometric residuals. We show results on real-world scenes and complex trajectories through simulated environments.
  arXiv  Detail & Related papers  (2024-10-11T12:01:15Z)
• HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction [46.269350101349715]
  HiSplat is a novel framework for generalizable 3D Gaussian Splatting. It generates hierarchical 3D Gaussians via a coarse-to-fine strategy. It significantly enhances reconstruction quality and cross-dataset generalization.
  arXiv  Detail & Related papers  (2024-10-08T17:59:32Z)
• Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos [58.22272760132996]
  We show that existing 4D Gaussian methods dramatically fail in this setup because the monocular setting is underconstrained. We propose Dynamic Gaussian Marbles, which consist of three core modifications that target the difficulties of the monocular setting. We evaluate on the Nvidia Dynamic Scenes dataset and the DyCheck iPhone dataset, and show that Gaussian Marbles significantly outperforms other Gaussian baselines in quality.
  arXiv  Detail & Related papers  (2024-06-26T19:37:07Z)
• SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation [55.2480439325792]
  We study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance.
  arXiv  Detail & Related papers  (2024-05-16T12:22:41Z)
• SplatMesh: Interactive 3D Segmentation and Editing Using Mesh-Based Gaussian Splatting [86.50200613220674]
  A key challenge in 3D-based interactive editing is the absence of an efficient representation that balances diverse modifications with high-quality view synthesis under a given memory constraint. We introduce SplatMesh, a novel fine-grained interactive 3D segmentation and editing algorithm that integrates 3D Gaussian Splatting with a precomputed mesh. By segmenting and editing the simplified mesh, we can effectively edit the Gaussian splats as well, which will lead to extensive experiments on real and synthetic datasets.
  arXiv  Detail & Related papers  (2023-12-26T02:50:42Z)
• FiGURe: Simple and Efficient Unsupervised Node Representations with Filter Augmentations [1.9922905420195374]
  This paper presents a simple filter-based augmentation method to capture different parts of the eigen-spectrum. We show that sharing the same weights across these different filter augmentations is possible, reducing the computational load. In addition, previous works have shown that good performance on downstream tasks requires high dimensional representations.
  arXiv  Detail & Related papers  (2023-10-03T08:54:06Z)
• Performance Embeddings: A Similarity-based Approach to Automatic Performance Optimization [71.69092462147292]
  Performance embeddings enable knowledge transfer of performance tuning between applications. We demonstrate this transfer tuning approach on case studies in deep neural networks, dense and sparse linear algebra compositions, and numerical weather prediction stencils.
  arXiv  Detail & Related papers  (2023-03-14T15:51:35Z)
• Fast Non-Rigid Radiance Fields from Monocularized Data [66.74229489512683]
  This paper proposes a new method for full 360deg inward-facing novel view synthesis of non-rigidly deforming scenes. At the core of our method are 1) An efficient deformation module that decouples the processing of spatial and temporal information for accelerated training and inference; and 2) A static module representing the canonical scene as a fast hash-encoded neural radiance field. In both cases, our method is significantly faster than previous methods, converging in less than 7 minutes and achieving real-time framerates at 1K resolution, while obtaining a higher visual accuracy for generated novel views.
  arXiv  Detail & Related papers  (2022-12-02T18:51:10Z)
• Condensing Graphs via One-Step Gradient Matching [50.07587238142548]
  We propose a one-step gradient matching scheme, which performs gradient matching for only one single step without training the network weights. Our theoretical analysis shows this strategy can generate synthetic graphs that lead to lower classification loss on real graphs. In particular, we are able to reduce the dataset size by 90% while approximating up to 98% of the original performance.
  arXiv  Detail & Related papers  (2022-06-15T18:20:01Z)
• The SKIM-FA Kernel: High-Dimensional Variable Selection and Nonlinear Interaction Discovery in Linear Time [26.11563787525079]
  We show how a kernel trick can reduce computation with suitable Bayesian models to O(# covariates) time for both variable selection and estimation. Our approach outperforms existing methods used for large, high-dimensional datasets.
  arXiv  Detail & Related papers  (2021-06-23T13:53:36Z)
• Tensor feature hallucination for few-shot learning [17.381648488344222]
  Few-shot classification addresses the challenge of classifying examples given limited supervision and limited data. Previous works on synthetic data generation for few-shot classification focus on exploiting complex models. We investigate how a simple and straightforward synthetic data generation method can be used effectively.
  arXiv  Detail & Related papers  (2021-06-09T18:25:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.