SLS4D: Sparse Latent Space for 4D Novel View Synthesis

• URL: http://arxiv.org/abs/2312.09743v1
• Date: Fri, 15 Dec 2023 12:31:20 GMT
• Title: SLS4D: Sparse Latent Space for 4D Novel View Synthesis
• Authors: Qi-Yuan Feng, Hao-Xiang Chen, Qun-Ce Xu, Tai-Jiang Mu
• Abstract summary: Existing dynamic NeRFs usually exploit a locally dense grid to fit the deformation field. We observe that the 4D space is inherently sparse. We propose to represent the 4D scene using a learnable sparse latent space, a.k.a. SLS4D.
• Score: 13.73892118198658
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural radiance field (NeRF) has achieved great success in novel view synthesis and 3D representation for static scenarios. Existing dynamic NeRFs usually exploit a locally dense grid to fit the deformation field; however, they fail to capture the global dynamics and concomitantly yield models of heavy parameters. We observe that the 4D space is inherently sparse. Firstly, the deformation field is sparse in spatial but dense in temporal due to the continuity of of motion. Secondly, the radiance field is only valid on the surface of the underlying scene, usually occupying a small fraction of the whole space. We thus propose to represent the 4D scene using a learnable sparse latent space, a.k.a. SLS4D. Specifically, SLS4D first uses dense learnable time slot features to depict the temporal space, from which the deformation field is fitted with linear multi-layer perceptions (MLP) to predict the displacement of a 3D position at any time. It then learns the spatial features of a 3D position using another sparse latent space. This is achieved by learning the adaptive weights of each latent code with the attention mechanism. Extensive experiments demonstrate the effectiveness of our SLS4D: it achieves the best 4D novel view synthesis using only about $6\%$ parameters of the most recent work.

Related papers

• Segment Any 4D Gaussians [69.53172192552508]
  We propose Segment Any 4D Gaussians (SA4D) to segment anything in the 4D digital world based on 4D Gaussians. SA4D achieves precise, high-quality segmentation within seconds in 4D Gaussians and shows the ability to remove, recolor, compose, and render high-quality anything masks.
  arXiv  Detail & Related papers  (2024-07-05T13:44:15Z)
• Self-Calibrating 4D Novel View Synthesis from Monocular Videos Using Gaussian Splatting [14.759265492381509]
  We propose a novel approach that learns a high-fidelity 4D GS scene representation with self-calibration of camera parameters. It includes the extraction of 2D point features that robustly represent 3D structure. Results show significant improvements over state-of-the-art methods for 4D novel view synthesis.
  arXiv  Detail & Related papers  (2024-06-03T06:52:35Z)
• Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models [116.31344506738816]
  We present a novel framework, textbfDiffusion4D, for efficient and scalable 4D content generation. We develop a 4D-aware video diffusion model capable of synthesizing orbital views of dynamic 3D assets. Our method surpasses prior state-of-the-art techniques in terms of generation efficiency and 4D geometry consistency.
  arXiv  Detail & Related papers  (2024-05-26T17:47:34Z)
• 4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes [33.14021987166436]
  We introduce 4DRotorGS, a novel method that represents dynamic scenes with anisotropic 4D XYZT Gaussians. As an explicit spatial-temporal representation, 4DRotorGS demonstrates powerful capabilities for modeling complicated dynamics and fine details. We further implement our temporal slicing and acceleration framework, achieving real-time rendering speeds of up to 277 FPS on an 3090 GPU and 583 FPS on a 4090 GPU.
  arXiv  Detail & Related papers  (2024-02-05T18:59:04Z)
• Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking [52.393359791978035]
  Motion2VecSets is a 4D diffusion model for dynamic surface reconstruction from point cloud sequences. We parameterize 4D dynamics with latent sets instead of using global latent codes. For more temporally-coherent object tracking, we synchronously denoise deformation latent sets and exchange information across multiple frames.
  arXiv  Detail & Related papers  (2024-01-12T15:05:08Z)
• Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields [54.482261428543985]
  Methods that use Neural Radiance fields are versatile for traditional tasks such as novel view synthesis. 3D Gaussian splatting has shown state-of-the-art performance on real-time radiance field rendering. We propose architectural and training changes to efficiently avert this problem.
  arXiv  Detail & Related papers  (2023-12-06T00:46:30Z)
• Tensor4D : Efficient Neural 4D Decomposition for High-fidelity Dynamic Reconstruction and Rendering [31.928844354349117]
  We propose an efficient 4D tensor decomposition method for dynamic scenes. We show that our method is able to achieve high-quality dynamic reconstruction and rendering from sparse-view camera or even a monocular camera. The code and dataset will be released atliuyebin.com/tensor4d-tensor4d.html.
  arXiv  Detail & Related papers  (2022-11-21T16:04:45Z)
• NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields [99.57774680640581]
  We present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering. We propose to decompose the 4D space according to temporal characteristics. Points in the 4D space are associated with probabilities belonging to three categories: static, deforming, and new areas.
  arXiv  Detail & Related papers  (2022-10-28T07:11:05Z)
• LoRD: Local 4D Implicit Representation for High-Fidelity Dynamic Human Modeling [69.56581851211841]
  We propose a novel Local 4D implicit Representation for Dynamic clothed human, named LoRD. Our key insight is to encourage the network to learn the latent codes of local part-level representation. LoRD has strong capability for representing 4D human, and outperforms state-of-the-art methods on practical applications.
  arXiv  Detail & Related papers  (2022-08-18T03:49:44Z)
• V4d: voxel for 4d novel view synthesis [21.985228924523543]
  We utilize 3D Voxel to model the 4D neural radiance field, short as V4D, where the 3D voxel has two formats. The proposed LUTs-based refinement module achieves the performance gain with little computational cost.
  arXiv  Detail & Related papers  (2022-05-28T04:45:07Z)
• 4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface [7.637832293935966]
  We introduce 4DComplete, a novel data-driven approach that estimates the non-rigid motion for the unobserved geometry. For network training, we constructed a large-scale synthetic dataset called DeformingThings4D.
  arXiv  Detail & Related papers  (2021-05-05T07:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.